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ABSTRACT 


The  systems  engineering  technical  review  (SETR)  process  should  be  an  event- 
driven  process  that  consists  of  one  or  more  programmatically  independent  reviews  with 
defined  entrance  and  exit  criteria.  The  SETR  assesses  the  technical  health,  requirements 
accuracy,  design  maturity,  testing  effectiveness,  and  sustainment  support  over  the 
program  life  cycle.  This  thesis  focuses  on  how  to  tailor  the  SETR  implementation  to 
improve  the  program’s  return  on  investment  (ROI).  To  address  the  research  question,  a 
literature  review  was  performed  focusing  on  SETR-related  policy,  instructions,  and 
memoranda  and  a  survey  was  conducted  of  subject  matter  experts.  Leveraging  a  tailored 
SETR  implementation  provides  the  necessary  structured  engineering  framework  while 
keeping  up  with  a  dynamic  software  development  environment  to  meet  the  increasing 
need  for  enhanced  capability  delivered  to  the  warfighter  in  a  shorter  timeframe.  More 
than  80%  of  the  survey  respondents  indicated  tailoring  was  occurring  within  programs  to 
address  specific  needs  such  as  leveraging  an  agile  software  development  model.  Factors 
external  to  the  organization  continue  to  be  the  primary  obstacle  no  matter  the  program 
size.  Future  naval  software  acquisition  programs  should  engage  leadership,  focus  on 
preparation,  maintain  communication  early  and  often,  and  educate  stakeholders  to 
improve  the  ROI  of  the  tailored  SETR  implementation. 
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EXECUTIVE  SUMMARY 


According  to  the  Naval  SYSCOM  Systems  Engineering  Policy  (2009),  the 
systems  engineering  technical  review  (SETR)  process  should  be  an  event-driven 
process  that  consists  of  one  or  more  programmatically  independent  reviews  with 
defined  entrance  and  exit  criteria.  The  SETR  reviews  assess  the  technical  health, 
requirements  accuracy,  design  maturity,  testing  effectiveness,  and  sustainment  support 
over  the  program  life  cycle.  The  jointly  signed  Systems  Command  (SYSCOM)  policy 
states  that  the  program  SETR  events  along  with  the  event  entrance  and  exit  criteria  are 
documented  in  the  Systems  Engineering  Plan  (SEP),  which  is  signed  by  the  program 
Milestone  Decision  Authority  or  other  designated  authority  based  on  the  program 
Acquisition  Category  level.  The  SYSCOM  policy  notes  that  event  closure  normally 
occurs  only  after  the  exit  criteria  has  been  met,  but  the  SETR  technical  review  board 
(TRB)  chair  must  concur  with  the  identified  action  items  along  with  any  plan  of  action 
and  milestones  and/or  mitigations.  According  to  the  policy,  the  SETR  output  ultimately 
informs  the  program  manager  (PM)  if  the  program  is  technically  ready  to  move  on  to 
the  next  phase  of  the  acquisition  process. 

From  a  policy  perspective,  the  Department  of  Defense  (DOD)  and  Department 
of  the  Navy  (DON)  directives  and  instructions  lay  the  systems  engineering  foundation 
for  the  naval  SYSCOM  guidance  regarding  SETR  implementation.  In  addition,  the 
Navy  has  emphasized  the  need  to  deliver  capability  versus  systems,  and  acquisition  is 
impacted  by  this  capability  vision  requiring  innovative  application  of  SETR  for  the 
system  of  systems  (SoS)  or  platform  level  efforts,  which  traditional  SETR  is  not  well 
structured  to  support.  This  thesis  provides  an  explanation  of  the  traditional  SETR 
process  implementation  within  the  various  naval  SYSCOMs.  Additionally,  this  thesis 
provides  an  overview  of  how  the  SYSCOMs  enable  major  acquisition  programs  and 
other  projects  to  tailor  the  SETR  implementation  to  fit  the  cost,  schedule,  and 
performance  addressing  program  variables  such  as  new  software  development 
methodologies  and  the  Navy’s  capability  delivery  vision.  This  research  includes 
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recommendations  on  how  to  increase  likelihood  of  a  program  return  on  investment 
(ROI)  for  a  tailored  SETR  implementation. 

Leveraging  a  tailored  SETR  implementation  provides  the  necessary  structured 
engineering  framework  while  keeping  up  with  a  dynamic  software  development 
environment  to  meet  increasing  need  for  enhanced  capability  delivered  to  the 
warfighter  in  a  shorter  timeframe.  However,  how  can  the  SETR  implementation  be 
tailored  to  most  efficiently  leverage  resources  and  minimize  schedule  impact  while 
addressing  the  acquisition,  technical,  and  policy/legal  requirements  within  naval 
software  acquisitions?  To  accomplish  this  research,  an  empirical  research  methodology 
was  leveraged  consisting  of  four  phases.  First,  a  literature  review  was  performed 
primarily  focused  on  SETR-related  policy  across  the  DOD,  DON,  and  naval 
SYSCOMs.  The  next  phase  consisted  of  an  electronic  survey  with  program  system 
matter  experts  (SME)  who  had  key  leadership  roles  within  various  sizes  of  programs. 
Next,  data  from  the  survey  responses  and  key  findings  from  the  literature  review  were 
analyzed  together  to  identify  commonalities  and  differences  as  it  relates  to  the  research 
questions.  Finally,  research  conclusions  and  recommendations  for  future  tailored  SETR 
implementations  were  addressed  as  well  as  any  suggestions  to  extend  this  thesis 
research. 

More  than  80%  of  the  survey  respondents  from  across  various  DON 
organizations  indicated  tailoring  was  occurring  within  programs  to  address  program 
specific  needs  such  as  aggressive  schedule  and  leveraging  an  agile  software 
development  model.  The  tailored  SETR  events  are  directly  impacting  the  program’s 
next  phase  through  adjustment  of  technical  design  and  other  key  program  variables.  In  a 
tailored  SETR  implementation,  the  programs  find  the  most  ROI  by  tailoring  entrance 
criteria,  exit  criteria,  and/or  which  specific  reviews  are  included.  This  holds  true  for 
both  commercial  off-the-shelf  and  new  development  software  acquisition  models. 
Factors  external  to  the  organization  continue  to  be  the  primary  obstacle,  no  matter  the 
program  size,  for  determining  the  ability  to  successfully  tailor  and  implement  the  SETR 
process.  Future  naval  software  acquisition  programs  should  engage  leadership,  focus  on 
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preparation,  maintain  early  and  often  communication,  and  educate  stakeholders  to 
improve  ROI  of  the  tailored  SETR  implementation. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

The  systems  engineering  technical  review  (SETR)  process  should  be  an  event 
driven  process  that  consists  of  one  or  more  programmatically  independent  review  events 
with  defined  entrance  and  exit  criteria  (Department  of  Navy  Space  and  Naval  Warfare 
Systems  Command  [DON  SPAWAR]  2009,  7).  The  SETR  reviews  assess  the  technical 
health,  requirements  accuracy,  design  maturity,  testing  effectiveness,  and  sustainment 
support  of  the  program  being  reviewed  over  its  life  cycle.  The  jointly  signed  Systems 
Command  (SYSCOM)  policy  states  that  program  SETR  events  along  with  the  event 
entrance  and  exit  criteria  are  documented  in  the  Systems  Engineering  Plan  (SEP),  which 
the  SEP  is  signed  by  the  program  Milestone  Decision  Authority  (MDA)  or  other  designated 
authority  based  on  the  program  Acquisition  Category  (ACAT)  level.  The  SYSCOM  policy 
notes  that  event  closure  normally  occurs  only  after  the  exit  criteria  has  been  met,  and  the 
SETR  technical  review  board  (TRB)  chair  concurs  with  the  identified  action  items  along 
with  any  plan  of  action  and  milestones  (POAM)  and/or  mitigations.  According  to  the 
policy,  the  output  of  a  SETR  event  ultimately  informs  the  program  manager  (PM)  whether 
the  program  is  technically  ready  to  move  on  to  the  next  phase  of  the  acquisition  process. 

The  SETR  process  is  intended  to  align  with  the  phases  of  the  acquisition  process 
although  the  alignment  varies  depending  on  the  acquisition  model  being  executed.  The 
models  are  detailed  in  the  Department  of  Defense  (DOD)  Operations  of  the  Defense 
Acquisition  System  (2015).  From  a  software  acquisition  perspective,  the  DOD  recognizes 
four  models  to  which  major  ACAT  programs  align,  which  are  software  intensive, 
incrementally  deploying  software,  accelerated  acquisitions,  or  a  hybrid  acquisition  with 
software  being  the  dominate  component  (Department  of  Defense  [DOD]  2015). 

A  traditional  SETR  process  can  include  upwards  of  12  types  of  reviews  such  as  a 
system  requirements  review  (SRR),  preliminary  design  review  (PDR),  critical  design 
review  (CDR),  and  test  readiness  review  (TRR).  Various  individuals  are  involved  in  these 
reviews  to  include  program  engineers,  technical  subject  matter  experts  (SME),  and  other 
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independent  stakeholders  to  ensure  their  areas  are  appropriately  addressed  within  the 
program  (DON  SPAWAR  2009).  For  example,  the  stakeholders  in  a  TRR  event  would 
include  individuals  from  the  developmental  and/or  operational  testing  agency.  Figure  1 
shows  the  timing  of  traditional  SETR  events  in  relation  to  the  program  acquisition  phases. 
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Program  Initiation  at  Milestone  A 


Figure  1.  Traditional  SETR  Model.  Source:  DON  SPAWAR 

(2009,  Enclosure  [1]). 


B.  PURPOSE 

Leveraging  a  tailored  SETR  implementation  provides  the  necessary  structured 
engineering  reviews  while  keeping  pace  with  a  dynamic  software  development 
environment  to  meet  the  increasing  need  for  enhanced  capability  delivered  to  the 
warfighter  in  a  shorter  timeframe.  In  addition,  the  Navy  has  emphasized  the  need  to 
deliver  capability  versus  systems,  and  acquisition  is  impacted  by  this  capability  vision 
requiring  innovative  application  of  SETR  for  the  system  of  systems  (SoS)  or  platform 
level  efforts,  which  tradition  SETR  is  not  well  structured  to  support.  This  thesis  provides 
an  explanation  of  the  traditional  SETR  process  implementation  within  the  various  DON 
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SYSCOMs  after  the  release  of  the  memorandum  from  the  Office  of  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  &  Acquisition)  (ASN[RDA])  in  June 
2008  directing  this  process  implementation.  Additionally,  this  thesis  will  provide  an 
overview  of  how  the  SYSCOMs  enable  major  acquisition  programs  and  other  projects  to 
tailor  their  processes  to  fit  the  cost,  schedule,  and  performance  to  address  program 
impacts  such  as  new  software  development  methodologies  and  the  Navy’s  capability 
delivery  vision.  This  research  includes  recommendations  on  how  to  increase  likelihood  of 
a  program’s  positive  return  on  investment  (ROI)  for  a  tailored  SETR  implementation. 

C.  RESEARCH  QUESTIONS 

The  primary  research  question  for  this  thesis  is:  How  can  the  SETR 
implementation  be  tailored  to  most  efficiently  leverage  resources  and  minimize  schedule 
impact  while  addressing  the  acquisition,  technical,  and  statutory  and  regulatory 
requirements  within  naval  software  acquisitions?  To  answer  this  question,  the  following 
subsidiary  questions  will  also  be  addressed. 

1.  What  are  the  obstacles  (e.g.,  policy,  timelines,  and  maturity)  and  risks  faced 
by  naval  programs  that  have  attempted  to  tailor  the  SETR  process?  Do  these 
obstacles  vary  based  on  the  size  of  the  program  (e.g.,  ACAT I  vs  non 
program  of  record)? 

2.  What  are  the  critical  elements  of  the  current  system  engineering  technical 
review  process  (e.g.,  entrance/exit  criteria,  order  of  reviews,  risk 
assessment)  that  program  managers  and/or  lead  engineers  should  include  in 
a  tailored  implementation?  What  incentives  or  return  on  investment  do  these 
critical  elements  provide? 

3.  Are  there  any  considerations/differences  that  should  be  accounted  for  when 
tailoring  the  process  to  support  various  acquisition  models[(e.g., 
commercial  off-the-shelf  (COTS),  new  development,  government  off-the- 
shelf  (GOTS)]? 

4.  What  can  engineers  and  program  managers  of  future  naval  acquisitions 
learn  from  other  programs  that  have  attempted  to  tailor? 

D.  BENEFIT  OF  STUDY 

This  research  includes  recommendations  based  on  lessons  learned  on  how  to 
increase  likelihood  of  a  program  ROI  for  a  tailored  SETR  implementation  as  it  relates  to 
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naval  software  acquisitions.  The  recommendations  will  assist  program  leadership  in 
making  better  decisions  on  where  to  allocate  software  engineering  resources  within  the 
schedule  and  funding  constraints.  While  the  thesis  is  focused  on  the  DON,  the 
recommendations  are  applicable  to  SETR  implementations  for  software  acquisitions 
programs  across  the  broader  DOD. 

E.  METHODOLOGY 

To  accomplish  this  research,  an  empirical  research  methodology  was  used  in  four 
phases.  First,  a  literature  review  was  performed  focused  on  SETR-related  policy, 
instructions,  and  memoranda  across  the  DOD,  DON,  and  naval  SYSCOMs.  The  literature 
review  also  included  other  documentation  (e.g.,  standards)  from  industry  organizations 
such  as  Institute  of  Electrical  and  Electronics  Engineers  (IEEE).  The  next  phase  consisted 
of  an  electronic  survey  with  program  SME  who  had  key  leadership  roles  (e.g.,  program 
manager,  lead  engineer)  within  various  sizes  of  programs  (e.g.,  ACAT  I,  project).  Table  4 
includes  the  survey  questions  along  with  the  potential  responses.  Next,  data  from  the 
survey  responses  and  key  findings  from  the  literature  review  were  analyzed  together  to 
identify  commonalities  and  differences  as  it  relates  to  the  research  questions.  Finally, 
research  conclusions  and  recommendations  specific  to  programs  who  wish  to  tailor  SETR 
implementation  were  addressed  as  well  as  some  suggestions  to  extend  this  thesis  research. 

F.  THESIS  ORGANIZATION 

The  chapters  of  the  thesis  are  organized  as  follows:  Chapter  1  is  the  introductory 
and  background  information.  Chapter  II  is  an  overview  of  the  SETR  implementation  which 
includes  potential  ways  in  which  the  process  could  be  tailored,  SETR  success  indicators, 
and  applying  SETR  to  support  program  risk  mitigation.  Chapter  III  is  a  review  of  the  DOD, 
DON,  naval  SYSCOM  policies,  and  industry  standards  relating  to  the  SETR  process. 
Chapter  IV  presents  the  data  from  the  survey.  Chapter  V  is  the  research  analysis  and 
discussion  tying  the  policy  review  presented  in  Chapter  III  to  the  survey  data  with  the  goal 
of  addressing  the  research  questions.  Chapter  VI  is  the  conclusions  from  the  research  and 
suggestions  for  follow  on  research. 
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II.  SYSTEMS  ENGINEERING  TECHNICAL  REVIEW 

IMPLEMENTATION 

A.  INTRODUCTION 

ASN  (RDA)  directed  in  a  13  June  2008  memorandum  that  the  SYSCOMs  develop 
and  implement  the  SETR  process  within  their  commands  no  later  than  120  days  from  the 
date  of  memorandum  to  “ensure  appropriate  systems  engineering  aspects  are  included  in 
the  Gate  Reviews”  (1).  The  SETR  process,  which  aligns  to  the  acquisition  phases,  has 
since  been  documented  in  various  SYSCOMs  instructions  and  other  supporting 
documentation  as  the  implementation  has  matured.  According  to  the  jointly  signed 
SYSCOM  policy  titled  Naval  SYSCOM  Systems  Engineering  Policy ,  SETRs  are  “event- 
driven,  have  specific  entry  and  closure  criteria,  and  are  used  to  evaluate  technical 
baselines.  The  SETRs  are  chaired  by  technical  authorities  independent  of  the  program, 
with  participation  by  the  PM  and  support  from  the  associated  Integrated  Product  Team 
(IPT)  (government/contractor)”  (DON  SPAWAR  2009,  7).  The  SYSCOM  policy  states 
the  SETR  process  assesses  the  program’s  technical  health,  requirements  accuracy,  design 
maturity,  testing  effectiveness,  and  sustainment  support  over  the  program  life  cycle.  The 
SETR  events  that  the  program  executes  along  with  the  event  entrance  and  exit  criteria  are 
documented  in  the  SEP,  which  is  signed  by  the  program  MDA  or  other  designated 
authority  based  on  the  program’s  AC  AT  level.  Closure  of  the  event  normally  occurs  only 
after  the  exit  criteria  has  been  met;  however,  the  SETR  TRB  chair  can  close  an  event 
with  outstanding  action  items  as  long  as  the  chair  concurs  with  the  identified  action  items 
along  with  any  POAM  and/or  mitigations.  According  to  the  joint  SYCOM  policy,  the 
output  of  a  SETR  event  ultimately  informs  the  PM  if  the  program  is  technically  ready  to 
move  on  to  the  next  phase  of  the  acquisition  process. 

The  Naval  Sea  Systems  Command  (NAVSEA)  Research  &  Systems  Engineering 
Warfare  Systems  Engineering  and  Human  Systems  Integration  Directorate  (05H) 
includes  this  comprehensive  list  of  SETR  objectives  in  the  December  2009  Technical 
Review  Manual: 


5 


Assessing  the  development  maturity  based  on  technology  maturity  and 
technical  development  goals  from  the  SEP,  SE  events  and 
accomplishments,  and  empirical  test  data  supporting  progress  to  date. 

Ensuring  operational,  functional,  performance,  Information  Assurance 
(IA)  and  cost  requirements,  designs,  technical  performance  measurements, 
and  technical  plans  are  being  tracked  and  are  on  schedule. 

Assessing  the  system  requirements  to  ensure  that  the  requirements  are 
unambiguous,  consistent,  complete,  feasible,  verifiable,  ranked  for  priority 
and  stability,  and  traceable  to  top-level  requirements  (TLRs). 

Demonstrating  that  the  relationships,  interactions,  interdependencies,  and 
interfaces  between  required  items  and  externally  interfacing  items,  system 
functions,  subsystems,  and  system  elements  (including  usability  and 
maintainability),  as  appropriate,  have  been  addressed. 

Ensuring  open  architecture  (OA)  in  the  emerging  system;  assessing  the 
degree  of  Naval  Enterprise  reuse  (availability  and  potential  for  component 
reuse);  and  critiquing  any  tradeoff  decisions. 

Ensuring  that  the  results  of  trade  studies  are  used  to  define  concepts  and 
that  risks  associated  with  alternatives  have  been  analyzed. 

Ensuring  that  technical  designs  will  be  usable  by  the  target  warfighter 
population,  meet  the  Fleet  requirements  and  have  viable  training  options. 

Ensuring  that  trade  studies  include  IA  and  safety  requirements  and  that  the 
risks  of  failing  to  meet  these  requirements  have  been  properly  treated. 

Confirming  that  the  effects  of  technical  risks  on  cost,  schedule,  and 
performance,  as  well  as  risk  reduction  measures,  rationale,  and 
assumptions  made  in  quantifying  the  risks  have  been  addressed. 

Providing  a  forum  for  communication,  coordination,  and  integration 
across  all  acquisition  disciplines. 

Establishing  a  common  configuration  baseline  from  which  to  proceed  to 
the  next  level  of  design. 

Confirming  that  continued  development  is  warranted  (with  or  without 
modifications  to  the  program),  and  when  it  is  not,  identifying  the  technical 
measures  necessary  to  preserve  for  reuse  as  much  of  the  technology, 
hardware  (HW),  and  software  (SW)  developed  to  date  as  possible.  In  the 
case  where  program  redirection  or  restructuring  is  considered  appropriate, 
the  review  should  ensure  that  executable  alternatives  have  been  defined 
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(discontinue  development,  or  take  corrective  action  on  the  products  and/or 
processes  of  the  current  phase  before  proceeding  to  the  next  phase). 

Verifying  the  system  is  ready  for  the  appropriate  level  and  type  of  testing 
or  that  appropriate  approvals  for  production  and  certification  have  been 
granted 

Identifying  resources  (e.g.,  people,  funding,  support  assets,  test  facilities, 
as  appropriate)  required  for  continued  development,  testing,  or  production. 

Reaching  technical  concurrence  with  stakeholders. 

Provide  a  check  of  proposed  design  configuration  versus  specification  and 
contractual  requirements. 

Evaluate  systems  configuration  at  different  stages  in  terms  of 
requirements.  (Department  of  Navy  Naval  Sea  Systems  Command  [DON 
NAVSEA]  2009,  3-2-3-3) 

To  accomplish  these  objectives,  various  individuals  are  involved  in  the  reviews  to 
ensure  their  areas  of  expertise  are  appropriately  addressed  within  the  program.  Key 
participant  roles  that  are  consistent  across  the  SYSCOM  implementations  include:  SETR 
chair(s),  independent  assessors,  and  integrated  product  team  (IPT)  participants.  The 
responsibilities  vary  at  the  SYSCOM  level  for  each  of  these  key  participant  roles,  but 
generally  only  to  account  for  their  organizational  construct  and  do  not  represent  a 
substantive  variation. 

The  SETR  TRB  chair  position  can  be  held  by  one  or  more  persons  if  there  is 
significant  interest  in  the  program  itself  and/or  perhaps  a  significant  partner  that  has 
interest  in  the  outcome  of  the  system  being  delivered.  Based  on  review  of  all  SYSCOMs 
SETR  policies,  the  minimum  responsibilities  of  the  chair(s)  include: 

•  concurring  on  the  entrance  and  exit  criteria  for  the  review 

•  facilitating  collaborative  participation  throughout  SETR  process 

•  providing  viable  technical  recommendation/guidance  to  address  risks  and 
issues  identified  in  the  process 

•  elevating  any  technical  issues  that  cannot  be  resolved  by  SETR 
participants  to  SYSCOM  Chief  Engineer  (CHENG) 

•  signing  the  SETR  report  (DON  SPA  WAR  2009,  9-12) 
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The  SETR  event  independent  assessors  are  either  a  Technical  Authority  (TA) 
related  to  the  area  being  covered  or,  at  a  minimum,  a  designated  representative  of  the 
SYSCOM  TA.  In  some  cases,  the  TA  might  also  be  asked  to  co-chair  the  SETR  event. 
From  a  responsibility  perspective,  the  independent  assessors  will  focus  their  efforts  on 
assessing  risk  in  their  area  of  expertise  as  well  as  providing  viable  technical 
recommendations/guidance  to  address  these  risks. 

Beyond  the  SETR  TRB  chair  and  independent  assessors,  the  IPT  participants  are 
key  members  of  the  SETR  review  team  and  make  up  the  bulk  of  the  membership 
especially  from  an  execution  of  work  perspective.  These  individuals  review  the  artifacts 
associated  with  the  SETR  event  to  validate  that  the  documents  align  to  the  higher  level 
naval  guidance  requirements,  do  not  conflict  with  one  another,  and  address  the  intent  of 
the  review  and  the  overall  program  (DON  SPAWAR  2016c).  The  risk  and  issues  are 
identified  on  the  basis  of  the  work  from  these  members,  which  will  ultimately  turn  into 
action  items  if  not  addressed  prior  to  the  closure  of  the  SETR  event  as  discussed  in  the 
SYSCOM  policy.  Figure  2  provides  a  SYSCOM  level  example  of  how  the  roles 
organizationally  interact.  Participants  include  a  TRB,  SETR  support  team  (SST),  and 
independent  assessment  team  (IAT).  The  participants  ultimately  report  back  to  the  PM 
and  CHENG. 
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Figure  2.  Example  SYSCOM  SETR  Organization  Chart. 

Source:  DON  SPA  WAR  (2016c,  8). 

The  primary  traditional  SETR  implementation  events  include  the  following: 

•  Critical  Design  Review  (CDR) 

•  Functional  Configuration  Audit 

•  Integration  Readiness  Review 

•  Operational  Test  Readiness  Review 

•  Physical  Configuration  Audit 

•  Preliminary  Design  Review  (PDR) 

•  Production  Readiness  Review 

•  Software  Specification  Review 

•  System  Functional  Review 

•  System  Requirements  Review 

•  System  Verification  Review 
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Test  Readiness  Review  (Department  of  Navy  Marine  Corps  Systems 
Command  [DON  MCSC]  2014). 


Each  event  aligns  to  a  different  acquisition  life  cycle  phase  and  determines  the  programs 
technical  readiness  to  proceed  into  the  next  phase.  Generally,  entrance  and  exit  criteria 
are  fulfillment  of  a  checklist  of  documentation  and  key  program  elements  (e.g., 
verification  of  completeness  of  test  procedures).  The  entrance  criteria  ensure  the  program 
is  sufficiently  technically  mature  to  enter  the  event.  SETR  events  are  resource  intensive, 
so  premature  entry  will  only  frustrate  all  participants  (at  best).  Similarly,  the  exit  criteria 
ensure  the  program  is  technically  ready  to  move  to  the  next  program  phase.  Examples  of 
both  entrance  and  exit  criteria  are  shown  in  Table  1. 


Table  1.  Example  SETR  Entrance  and  Exit  Criteria. 

Source:  DON  SPAWAR  (2016c,  48). 
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B.  PROCESS  TAILORING 


Each  SYSCOM  has  varying  guidance  regarding  tailoring.  Some  SYSCOMs  only 
lightly  touch  on  the  subject  requiring  that  the  tailoring  is  documented  and  agreed  to 
within  the  program’s  SEP,  while  others  provide  more  specific  models  and  guidance.  For 
example,  SPAWAR’s  Systems  Engineering  Technical  Review  Organizational  Standard 
Process  Handbook  provides  specific  tailored  SETR  models  such  as  the  Accelerated 
Acquisition  Model.  Similarly,  but  in  the  commercial  sector,  the  IEEE  Standard  15288, 
which  is  titled  Systems  and  software  engineering — Systems  life  cycle  processes,  has  an 
appendix  focused  on  process  tailoring  that  is  extremely  applicable  to  many  DOD 
acquisition  programs.  IEEE  Standard  15288.2,  which  is  titled  Technical  Reviews  and 
Audits  on  Defense  Programs ,  gets  into  SETR  events  and  amplifies  IEEE  15288.  When 
determining  if  tailoring  the  SETR  implementation  is  beneficial,  it  is  important  to  be  able 
to  articulate  the  purpose,  outcomes,  and  activities/tasks  associated  with  the  tailoring, 
which  are  the  key  points  of  IEEE  Standard  15288  Annex  A.  With  regards  to  the 
activities,  the  program  should  be  able  address  the  following  items  detailed  in  IEEE 
Standard  15288: 

Identify  and  record  the  circumstances  that  influence  tailoring.  These 
influences  include,  but  are  not  limited  to: 

Stability  of,  and  variety  in,  operational  environments 

Risks,  commercial  or  performance,  to  the  concern  of  interested 

parties; 

Novelty,  size  and  complexity; 

Starting  date  and  duration  of  utilization; 

Integrity  issues  such  as  safety,  security,  privacy,  usability, 

availability; 

Emerging  technology  opportunities; 

Profile  of  budget  and  organizational  resources  available; 

Availability  of  the  services  of  enabling  systems; 
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Roles,  responsibilities,  accountabilities  and  authorities  in  the 

overall  life  cycle  of  the  system; 

The  need  to  conform  to  other  standards. 

In  the  case  of  properties  critical  to  the  system,  take  due  account  of  the  life 
cycle  structures  recommended  or  mandated  by  standards  relevant  to  the 
dimensions  of  the  criticality. 

Obtain  input  from  parties  affected  by  the  tailoring  decisions.  This 
includes,  but  may  not  be  limited  to: 

The  system  stakeholders; 

The  interested  parties  to  an  agreement  made  by  the  organization; 

The  contributing  organizational  functions. 

Make  tailoring  decisions  in  accordance  with  the  Decision  Management 
process  to  achieve  the  purpose  and  outcomes  of  the  selected  life  cycle 
model. 

Select  the  life  cycle  process  that  require  tailoring  and  delete  selected 
outcomes,  activities,  or  tasks.  (IEEE  Computer  Society  2015,  86-87) 

These  activities  are  consistent  with  the  principles  within  the  SYSCOMs’  guidance 

as  well.  For  instance,  SPAWAR’s  Systems  Engineering  Technical  Review  Organizational 

Standard  Process  Handbook  defines  tailoring  as  that  which  is  “based  on  sound  systems 

engineering  practices,  program  or  project  complexity,  and  risk  level,  while  meeting  the 

objectives  of  the  identified  event”  (2016c,  4).  This  definition  ties  closely  to  the  IEEE 

Standard  15288  “identification  and  recording  of  the  circumstances  that  influence  the 

tailoring”  as  well  as  defining  the  purpose  and  outcome  of  tailoring  the  process  itself 

(IEEE  Computer  Society  2015,  86).  NAVSEA’s  05H  Technical  Review  Manual  directs 

that  tailoring  be  “consistent  with  good  engineering  judgment  and  program  complexity 

and  risk  levels”  (2009,  3-6).  Naval  Air  Systems  Command  (NAVAIR)  Instruction 

4355. 19E,  which  focuses  on  the  Systems  Engineering  Technical  Review  Process, 

reinforces  similar  concepts  to  those  of  both  SPAWAR  and  NAVSEA  do;  however, 

NAVAIR  goes  a  step  further  to  explain  that  “tailoring  takes  the  form  of  deletion  (removal 

of  reviews  and  elements  not  applicable),  alteration  (modifying  and  combining  reviews 

and  elements  to  more  explicitly  reflect  the  application  to  a  particular  effort)  or  addition 
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(adding  reviews  and  elements  to  satisfy  program  elements)”  (2015,  6-7).  Even  with  the 
additional  context,  this  NAVAIR  Instruction  4355. 19E  is  still  very  consistent  with  the 
IEEE  Standard  15288  process  tailoring. 

C.  METRICS 

The  International  Council  on  Systems  Engineering  (INCOSE)  published  a 
Systems  Engineering  Leading  Indicators  Guide  on  29  January  2010,  which  defines  a 
leading  indicator  as  “a  measure  for  evaluating  the  effectiveness  of  how  a  specific  activity 
is  applied  on  a  project  in  a  manner  that  provides  information  about  impacts  that  are  likely 
to  affect  the  system  performance  objectives”  (6).  This  is  a  critical  definition  at  the  heart 
of  SETR  events  as  the  technical  leadership  of  the  program  should  be  looking  at  any 
leading  indicators  (specific  sets  of  metrics)  across  the  program  that  would  indicate 
impacts  that  would  likely  affect  the  systems  performance  or  ability  to  meet  the 
performance  objectives  laid  out  in  the  system  requirements  or  other  core  documentation 
(e.g.,  Capability  Development  Document).  Appendix  A  gives  additional  details  regarding 
sample  leading  indictors  based  on  IEEE  Standard  15288. 

Leading  indicators  generally  are  more  focused  on  predictive  analysis  to  support 
technical  leadership  decisions.  However,  it  is  important  to  not  ignore  conventional 
measures  within  the  SETR  event  to  ensure  the  technical  report,  which  is  an  output  of  the 
SETR,  provides  a  holistic  look  at  the  health  and  includes  best  recommendations  to  the 
PM.  Table  2  includes  some  sample  metrics  that  are  included  in  SPAWAR’s  Systems 
Engineering  Technical  Review  Organizational  Standard  Process  Handbook.  Several  of 
the  metrics  are  trending  related  around  tracking  risk  and  issues  (R&I)  (e.g.,  number  of 
R&I  by  program)  that  would  be  measured  over  multiple  SETR  events.  Using  metrics 
such  as  those  in  the  handbook  provide  a  continuous  process  improvement  aspect  to  the 
SETR  event  in  order  to  minimize  the  potential  of  it  being  executed  as  a  simple  check  in 
the  box.  In  addition,  the  metrics  enable  a  mechanism  to  assess  maximum  value  from  the 
program  office  perspective  and/or  a  retrospective  look  back  to  improve  in  the  next  review 
event  if  the  event  did  not  provide  value  relative  to  the  resources  expended. 
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Table  2.  Sample  Metrics.  Source:  DON  SPAWAR  (2016c,  7). 


Quantitative 

Trends 

Demand  Signal 

Number  of  SETR  by 

Type,  Maturity,  and 

PMW 

Number  of  R&I  by 

PEO,  PMW,  Program 

Workload  forecast 

Number  of  R&I  by 

Closed,  Open  with 
Mitigation,  Accepted 
Risk/Issue  with  No 
Further  Action 

By  Assessment  Area 

Loading 

Number  of  artifacts 
provided  initially 

Number  of  R&I  by 

SETR  Type 

Planned  events  vs 

Actual  events  as 

documented  in 

IMS,  SEPs 

Number  of  artifacts 
provided  after  initial 
upload  to  TRP 

Number  of  R&I 

elevated  to  RFA 

Specialized 
reviewers  needed 

Number  of  artifacts 
requested  additionally 
during  review  period 

Number  of  R&I  closed 
during  review 

Additionally,  ASN(RDA)’s  Guidebook  for  Acquisition  of  Naval  Software 
Intensive  Systems  published  in  September  2008  highlights  the  SETR  process  as  a  risk 
management  tool  as  it  can  be  used  to  verify  the  following: 

Requirements  for  the  acquired  system  are  defined  (including  granularity  to 
address  software- specific  requirements); 

Requirements  are  transformed  into  an  effective  system  (including  effective 
software  components  and  subsystems); 

The  processes  are  consistent  and  repeatable  for  the  entire  life  cycle,  where 
necessary; 

The  provider/supplier  uses  a  systematic  approach  in  providing  the  required 
products  and  services; 

The  test  resources  (personnel,  facilities,  test  assets,  test  plans,  ranges,  etc.) 
are  available  and  the  product  is  ready  for  test; 

The  product  is  ready  for  Operational  Evaluation  (OPEVAL)  (validation 
environment,  Operational  Test  &  Evaluation  Force  (OPTEVFOR) 
personnel,  ranges,  etc.); 
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Services/products  are  sustainable  throughout  the  life  cycle  (including  an 
adequate  software  support  activity);  and 

Systems  are  properly  disposed  of  when  they  are  retired  from  service. 

(Office  of  the  Assistant  Secretary  of  the  Navy  [Research,  Development  & 
Acquisition]  2008a,  6-10) 

Each  of  these  areas  highlighted  in  the  ASN(RDA)  guide  are  key  areas  that  the  IPT 
participants  or  SMEs  would  be  evaluating  as  they  review  various  documents  that  are  part  of 
the  entrance  criteria,  exit  criteria,  and/or  other  documents  that  are  requested  as  part  of 
requests  for  information  (RFI)  or  requests  for  action  (RFA)  during  the  event  itself.  By  tying 
the  leading  indicators,  metrics  and  risk  mitigations  together  a  program  can  ensure  the 
reviews  provide  measurable  ROI  to  the  program  office  as  it  moves  forward  in  delivering 
new  capability  to  the  warfighter. 

D.  SUMMARY 

ASN  (RDA)  Memorandum  from  13  June  2008,  which  is  titled  Systems 
Engineering  Technical  Review  Process  for  Naval  Acquisition  Programs,  directed  the 
SYSCOMs  develop  and  implement  the  SETR  process  within  their  commands  no  later 
than  120  days  from  the  date  of  memorandum.  The  SETR  process,  which  aligns  to  the 
acquisition  phases,  has  since  been  documented  in  various  SYSCOMs  instructions  and 
other  supporting  documentation  as  the  implementation  has  matured.  The  SETR  process 
should  be  an  event-driven  process  that  consists  of  one  or  more  independent  review  events 
to  assess  the  program’s  technical  health,  requirements  accuracy,  design  maturity,  testing 
effectiveness,  and  sustainment  support  over  the  program  life  cycle  (DON  SPAWAR 
2009,  7).  The  selection  of  SETR  events  to  include  any  intended  tailoring  that  the  program 
executes  is  documented  in  the  SEP.  Per  the  jointly  signed  SYSCOM  policy,  the  output  of 
a  SETR  event  ultimately  informs  the  PM  if  the  program  is  technically  ready  to  move  on 
to  the  next  phase  of  the  acquisition  process.  The  following  chapter  provides  a  review  of 
the  DOD,  DON,  naval  SYSCOM  policies,  and  industry  standards  that  give  direction  and 
guidance  related  to  the  process,  entrance/exit  criteria,  leadership  involvement,  tailoring, 
and  all  other  applicable  aspects  of  the  SETR  process. 
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III.  POLICY  AND  STANDARD  REVIEW 


A.  INTRODUCTION 

This  section  provides  a  review  of  the  DOD,  DON,  naval  SYSCOM  policies,  and 
industry  standards  that  give  direction  and  guidance  related  to  the  process,  entrance 
criteria,  exit  criteria,  leadership  involvement,  tailoring,  and  all  other  applicable  aspects  of 
the  SETR  process.  Each  policy  or  standard  will  be  summarized  with  an  overview  of  how 
it  relates  to  the  SETR  process,  highlight  any  strengths  of  the  document,  and  what  if  any 
possible  unanticipated  outcomes  could  be  triggered  by  the  policy  or  standard  itself.  Table 
3  includes  the  list  of  items  that  will  be  reviewed  as  part  of  this  section. 


Table  3.  Policy,  Instructions,  Memos  Review  List 


DOD  Policy,  Instructions,  &  Memos 

The  Defense  Acquisition  System  (DODD  5000.01) 

Operations  of  the  Defense  Acquisition  System  (DODI  5000.02) 

Business  System  Requirements  and  Acquisition  (  DODI  5000.75) 

Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  (DODI  5134.16) 

DON  Policy,  Instructions,  &  Memos 

ASN  RD&A  Memo,  SETR  Process  for  Naval  Acquisition  Programs,  13  June  08 

Naval  SYSCOM  Systems  Engineering  Policy  (SPAWARINST  5000.1) 

DON  Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities 
Integration  and  Development  System  (SECNAVINST  5000. 2E) 

SYSCOMs  Policy,  Instructions,  &  Memos 

NAVSEA  05H  Technical  Review  Manual,  Version  2.0,  18  December  2009 

NAVAIR  Systems  Engineering  Technical  Review  Process  (NAVAIRINST  4355. 19E) 

NAVAIR  Adapting  Acquisition  to  Agile  Software  Development:  A  How-To  Guide,  Version  2.0,  19 
March  2014 

MCSC  SETR  Handbook  (SIAT-HDBK-001) 

SPAWAR  Systems  Engineering  Policy  (SPAWARINST  5401.4) 

SPAWAR  Systems  Engineering  Technical  Review  Policy  (SPAWARINST  5400. 3A) 

SPAWAR  Systems  Engineering  Technical  Review  Organizational  Standard  Process  Handbook 

Industry  Standards,  Instructions,  &  Memos 

IEEE  Technical  Reviews  and  Audits  for  Defense 

SEI  Agile  Software  Teams:  How  They  Engage  with  Systems  Engineering  on  DOD  Acquisition 
Programs 
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B.  DEPARTMENT  OF  DEFENSE 


DOD  Directive  5000.01,  which  is  The  Defense  Acquisition  System  directive, 
establishes  the  basic  policy  governing  how  the  DOD  operates  from  an  acquisition 
perspective  to  include  responsibilities  for  key  officials  (e.g.,  the  Assistant  Secretary  of 
Defense,  Director  of  Operational  Test  and  Evaluation),  defining  key  terminology  (e.g., 
program  manager,  milestone  decision  authority),  and  outlining  policy  around  the  Defense 
Acquisition  System  (e.g.,  flexibility,  innovation)  (DOD  2007).  A  key  SETR-related 
element  in  this  instruction  is  directing  that  the  systems  engineering  approach  should 
optimize  “total  system  performance  and  minimize  total  ownership  costs”  (9). 
Fundamentally,  the  SETR  process  supports  the  key  tenants  of  this  systems  engineering 
policy,  so  it  is  incumbent  upon  all  involved  in  planning  and  executing  the  program  SETR 
to  ensure  the  reviews  do  not  become  a  “check-the-box”  event. 

DOD  Instruction  5000.02,  which  is  the  Operation  of  the  Defense  Acquisition 
System  instruction,  establishes  the  policy  for  how  acquisition  programs  shall  be  managed. 
This  is  the  first  DOD  policy  from  a  policy  hierarchy  perspective  where  SETR  reviews 
begin  to  appear.  Based  on  the  AC  AT  of  the  program,  enclosure  3  of  this  instruction  lays 
out  the  minimum  technical  reviews  that  program  managers  will  conduct,  which  consist  of 
a  PDR  and  a  CDR  (DOD  2015).  The  instruction  enclosure  also  includes  at  what  level 
these  reviews  will  be  conducted.  For  example  per  the  instruction,  MDAPS  can  conduct 
system  level  PDR  assessments  at  the  MDA  level  while  an  ACAT  IC  must  conduct  the 
review  at  the  Component  Acquisition  Executive  (CAE)  level. 

DOD  Instruction  5000.75,  which  is  the  Business  Systems  Requirements  and 
Acquisition  instruction,  became  effective  on  2  February  2017  during  this  research  effort. 
This  instruction  establishes  the  policy  for  how  business  systems  will  be  managed  (DOD 
2017).  This  instruction  supersedes  DODI  5000.02  for  “business  system  acquisition 
programs  that  are  not  designated  as  a  Major  Defense  Acquisition  Program  according  to 
DoDI  5000.02”  (1).  A  business  system  as  described  by  this  instruction  include  “financial 
systems,  financial  data  feeder  systems,  contracting  systems,  logistics  systems,  planning 
and  budgeting  systems,  installations  management  systems,  human  resources  management 

systems,  and  training  and  readiness  systems”  (31).  The  instruction  does  not  include 
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SETR-specific  language,  but  does  require  technical  and  management  assessments  to  be 
addressed  in  the  required  implementation  plan.  The  “assessment  structure  will  reflect  the 
program’s  tailored  software  development  life  cycle  and  may  include  events  such  as 
design  reviews  and  test  readiness  reviews  or  short  iteration  retrospectives  and  acceptance 
reviews”  (24).  The  SETR  related  policy  and  other  documentation  at  the  DON  and  naval 
SYSCOM  level  was  written  prior  to  this  instruction,  so  expectation  is  that  future 
iterations  of  these  documents  would  address  items  such  as  SETR  best  practices  and 
implications  related  to  this  instruction. 

DOD  Instruction  5134.16,  which  is  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering  (DASD(SE))  instruction,  primarily  establishes  the  roles  and 
responsibilities  of  the  DASD(SE)  position.  However,  this  instruction  from  a  SETR 
perspective  lays  the  foundation  for  the  SEP  including  who  needs  a  SEP  and  minimum 
items  that  should  be  covered  (e.g.,  reliability  growth,  considerations  for  life  cycle 
management  and  sustainability).  This  document  is  not  overly  prescriptive  by  outlining  the 
minimum  SETR  items  required  in  a  SEP,  which  are  expected  across  DOD  from  a 
consistency  standpoint. 

C.  DEPARTMENT  OF  NAVY 

ASN  (RDA)  Memorandum  from  13  June  2008,  which  is  titled  Systems 
Engineering  Technical  Review  Process  for  Naval  Acquisition  Programs,  directed  the 
naval  SYSCOMs  to  develop  and  implement  the  SETR  process  within  their  commands  no 
later  than  120  days  from  the  date  of  memorandum  to  “ensure  appropriate  systems 
engineering  aspects  are  included  in  the  Gate  Reviews”  (1).  The  2008  ASN  (RDA) 
memorandum  also  lays  out  other  fundamental  pieces  of  the  process  that  will  not  change 
such  as  having  including  independent  assessors,  extended  IPT  members,  and  enabling 
tailoring  based  on  program  scope  and  complexity.  From  a  naval  prospective,  this  is  a 
crucial  piece  of  SETR  documentation  that  lays  the  foundation,  from  which  all  the 
SYSCOMs’  SETR  frameworks  are  derived. 

SPAWARINST  5000.1,  which  is  the  Naval  SYSCOM  Systems  Engineering  Policy, 
SYSCOM  Engineering  and  Technical  Authority  Policy,  is  a  jointly  signed  naval  SYSCOM 
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policy  that  establishes  systems  engineering  policy  and  a  common  SETR  process  across  the 
DON  (DON  SPAWAR  2009).  From  a  background  perspective,  the  policy  baselines  roles, 
responsibilities,  and  definitions  previously  covered  in  other  documents  such  as  what  the 
PM’s  role  is,  what  systems  engineering  means,  and  what  the  acquisition  life  cycle  is.  The 
policy  states  the  SEP  must  detail  the  SETR  schedule,  which  will  help  in  understanding  the 
sequencing  relative  to  other  program  events.  Once  the  SEP  is  approved,  this  will  also 
indicate  approval  of  the  SETR  schedule.  In  the  section  specific  to  SETR  policy, 
SPAWARINST  5000.1  highlights  the  following  fundamental  SETR  elements: 

•  TRB  Chair:  must  be  designated  in  writing;  generally  a  senior  individual  in 
the  SYSCOM  technical  authority;  holds  the  final  authority  for  closing  the 
review 

•  Agenda:  includes  TRB  membership,  SETR  participants,  schedule,  and 
entrance/exit  criteria 

•  Conduct:  New  issues  should  not  arise  at  the  SETR  or  this  perhaps 
indicates  that  the  SETR  is  being  held  prematurely 

•  Document  Review:  depends  on  objective  analyses,  correctness  and 
completeness,  which  should  be  measured  against  clearly  stated  objectives 

•  Results:  signed  report  by  TRB  chair  closes  out  the  event —  must  capture 
action  items  with  appropriate  status  and  include  confidence  level  of 
baseline  to  move  forward  to  next  stage  of  development  (10-12) 

SPAWARINST  5000.1  stays  at  a  high  enough  level  of  policy  to  not  hinder  the  individual 
SYSCOMs  as  they  define  their  SETR  frameworks  and  guidance  to  the  acquisition  and 
technical  workforce.  This  document  also  provides  baseline  consistency  and  a  common 
vocabulary  for  how  SETRs  are  implemented  across  the  DON,  which  is  critical  as  the 
Navy  drives  towards  more  SoS  deliveries  requiring  more  cross  SYSCOM  interaction  and 
program  to  program  interdependencies. 

SECNAVINST  5000. 2E,  which  is  titled  Department  of  the  Navy  Implementation 
and  Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities  Integration  and 
Development  System,  establishes  the  procedures  for  defense  acquisition  programs 
specifically  related  to  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
(Office  of  the  Secretary  of  the  Navy  2011).  From  a  SETR  perspective,  the  instruction 
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provides  the  basis  for  the  concepts  mentioned  in  the  jointly  signed  SYSCOM  policy 
SPAWARINST  5000.1,  but  it  does  reach  a  much  broader  audience  being  a  SECNAVINST. 
Additionally,  the  instruction  introduces  a  new  “IT  Box”  concept  that  increases  the  value  of 
the  ability  to  tailor  a  SETR  implementation.  The  “IT  Box”  approach,  which  is  applied  to 
software  development  programs,  is  “meant  to  lighten  the  burden  of  JCIDS  as  the  program 
integrates  systems  enhancements  described  by  the  CDD  [Capabilities  Development 
Document],  and  allows  programs  to  take  full  advantage  of  evolving  commercial 
technologies”  (1-11).  In  lieu  of  writing  a  Capabilities  Production  Document,  the  “IT  Box” 
allows  programs  to  write  an  annex  to  the  CDD  or  an  existing  document  to  address  four 
critical  elements:  “definition  of  threshold  capability  levels  based  upon  today’s  technology, 
a  defined  process  for  oversight  and  approval  of  future  evolution,  a  defined  plan  for 
delivering  those  capabilities,  and  a  defined  level  of  funding”  (1-11).  The  tailoring  aspect  of 
SETR  is  critical  for  those  programs,  which  are  leveraging  “IT  Box,”  in  order  to  not  slow 
down  the  increased  design  to  deployment  schedule  enabled  by  “IT  Box.” 

D.  SYSTEMS  COMMANDS 

NAVSEA  05H  Technical  Review  Manual  published  in  December  2009  describes 
the  SETR  implementation  objectives  and  execution  guidelines  (e.g.,  scheduling, 
documentation).  The  appendices  of  the  document  go  into  detailed  descriptions  of 
entrance  and  exit  criteria  for  each  SETR  event,  different  documentation,  architectural 
views,  and  scheduling  details.  Overall,  the  document  is  the  most  comprehensive  of  the 
SYSCOMs’  SETR-related  documents  with  regards  to  execution  of  a  traditional  SETR. 
From  a  roles  and  responsibilities  perspective,  the  NAVSEA  SETR  TRB  chair,  per  the 
manual,  consists  of  a  minimum  two  co-chairs,  which  is  a  bit  different  than  the  other 
SYSCOMs.  The  co-chair  consists  of  the  PM  and  the  independent  TA,  which  can  be 
joined  by  other  co-chairs,  if  there  is  a  significant  partner  that  warrants  the  position  as 
detailed  in  the  manual.  The  responsibilities  across  the  chairs  are  still  consistent  with  what 
was  previously  described,  just  shared  across  all  the  chairs.  The  remainder  of  the  SETR 
organization  is  also  consistent  with  the  previously  described  organization  in  the  process 
overview.  The  NAVSEA  05H  Technical  Review  Manual  goes  into  great  detail  about 

initial  planning  for  closing  out  the  SETR  event,  which  is  depicted  at  a  high  level  in 
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Figure  3  from  the  manual.  These  steps  are  repeated  for  each  individual  review  (e.g.,  PDR, 
CDR)  within  the  SETR  implementation.  The  manual  also  includes  opportunities  and/or 
reminders  throughout  that  the  implementation  should  be  tailored  based  on  the  scope  and 
complexity  of  the  program,  but  it  does  not  give  any  models  on  how  to  do  this  as 
SPAWAR  does,  for  example.  This  provides  maximum  level  flexibility  for  program 
engineers  with  previous  experience  tailoring  and/or  motivation  to  seek  out  the 
opportunity.  However,  by  not  providing  any  best  practices,  models,  and/or  lessons 
learned,  it  allows  programs  to  each  independently  make  the  same  mistake,  which  could 
be  costly  in  time,  material,  and/  funding  to  the  DON. 


(TIM*)_ 


Plan 

•  Identify  participants 

•Assign  roles  and 
tasks 


Familiarize 

•  Have  overview 
meeting 


Pre-review 


•  Review  entrance 
criteria 


•  Examine  data 

•  Analyze  data 


Review 

•Individual  and 
team  reports  on 
review  findings 

•  Facilitate  and 
pace  meeting 

•  Examine  review 
data  and 
anaiyses- 
record  and 
classify  findings 


•Assign 

responsibility 


Follow-up 


•Track  RFA/ 

RFI  completion 
trends 

•  Document  and 
distribute  results 
of  review 
and  RFI/RFA 
completions 


Establish  guidelines 
and  procedures 

Establish  and  use 
entry  criteria 


Track  and 
document 
analysis 


Address  key 
issues  identified 
by  pre-review 
activity 


Establish  exit 
criteria  based  on  the 
event-driven 
schedule 


Assess  severity 
of  problems 

Identify  RFA/ 
RFIs  y 


•  Assess  exit  criteria 


Figure  3.  Sample  SETR  Execution. 

Source:  DON  NAVSEA  (2009,  3-15). 


NAVAIRINST  4355. 19E,  which  is  titled  Systems  Engineering  Technical  Review 
Process,  establishes  the  policy,  guidance,  and  responsibilities  with  respect  to  the  SETR 
implementation  for  the  command  (Department  of  Navy  Naval  Air  Systems  Command 
[DON  NAVAIR]  2015).  The  instruction,  which  was  published  in  February  2015,  starts 
very  early  introducing  a  new  SETR  event  called  the  Release  Backlog  Review  (RBR), 
which  NAVAIR  introduced  to  deal  with  the  agile  software  development  methodology. 
Later  in  the  instruction  enclosure,  NAVAIR  includes  entrance  and  exit  criteria 
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expectations  for  incorporating  this  event  into  the  SETR  implementation.  In  addition,  the 
NAVAIR  instruction  includes  an  entire  section  on  SETR  tailoring  explaining  that 
“tailoring  takes  the  form  of  deletion  (removal  of  reviews  and  elements  not  applicable), 
alteration  (modifying  and  combining  reviews  and  elements  to  more  explicitly  reflect  the 
application  to  a  particular  effort)  or  addition  (adding  reviews  and  elements  to  satisfy 
program  elements)”  (6-7).  Beyond  this  the  NAVAIR  instruction  follows  the  general 
principles  described  in  the  SETR  process  overview,  but  the  instruction  takes  a  strong 
stance  on  encouraging  tailoring  when  there  is  an  “opportunity  to  optimize  the  program 
execution  in  the  context  of  cost,  schedule,  and  performance”  (6). 

NAVAIR’ s  Adapting  Acquisition  to  Agile  Software  Development:  A  How-to 
Guide ,  which  was  published  in  March  2014,  documents  some  of  the  early  concepts  that 
were  formalized  in  the  NAVAIRINST  4355. 19E.  The  guide  gives  more  details  in  how  to 
implement  the  RBR  with  some  of  the  more  typical  SETR  events  such  as  PDR  and  CDR. 
RBR  generally  keeps  a  more  consistent  pace  with  what  agile  development  refers  to  as 
development  sprints,  which  require  government  SMEs  to  stay  closely  engaged.  At  some 
point  after  generally  multiple  RBR  have  been  executed,  a  PDR  will  be  held  that  will 
consist  of  the  software  architecture  from  these  executed  RBR  and  the  “tentative  allocated 
backlog  will  be  established”  (DON  NAVAIR  2014,  7).  RBR  will  continue  to  be  held 
until  the  appropriate  time  for  the  CDR  and  likewise  the  TRR,  which  is  shown  in  Figure  4. 
Generally,  NAVAIR  programs  are  either  Major  Defense  Acquisition  Programs  or  Major 
Automated  Information  Systems  including  both  HW  and  SW,  which  require  the  PDR  and 
CDR.  Otherwise,  the  program  could  tailor  the  SETR  implementation  further. 


Figure  4.  RBR  within  SETR  Implementation. 

Source:  DON  NAVAIR  (2014,  7). 
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Marine  Corps  Systems  Command  (MCSC)  SIAT-HDBK-001,  which  is  titled 
Systems  Engineering  Technical  Review  (SETR)  Handbook ,  provides  guidance  on  planning 
and  execution  of  SETR  events.  The  handbook  sets  the  tone  in  the  early  stages  of  the 
document  with  a  concise  graphic  that  shows  tailoring  opportunities  as  it  aligns  to  the 
entrance  into  the  acquisition  life  cycle  and  program  risk,  which  is  included  in  Figure  5 
(DON  MCSC  2014).  It  introduces  a  new  event  that  the  other  SYSCOMs  do  not  have, 
which  is  the  non-developmental  item  (NDI)  integration  review  (NIR).  At  a  high  level,  the 
intent  of  the  NIR,  per  the  handbook,  is  to  ensure  that  the  system  under  review  can  meet  the 
stated  performance  requirements  and  move  to  the  next  step  in  the  acquisition  life  cycle 
within  cost  and  schedule  while  integrating  the  COTS,  GOTS,  or  other  proposed  NDI.  This 
review  is  held  instead  of  a  PDR  and  CDR,  but  addresses  the  same  technical  rigor  of  these 
reviews.  The  MCSC  SETR  organizational  structure  as  well  as  planning,  execution,  and 
close  out  processes  are  similar  to  those  discussed  in  the  SETR  process  overview.  The 
MCSC  SETR  Handbook  has  a  detailed  appendix  for  each  SETR  event  that  includes  an 
event  overview,  entrance  criteria,  review  elements,  agenda,  exit  criteria,  and  evaluation 
criteria.  This  resource  acts  as  a  job  aid  and  future  reference  as  the  MCSC-related  programs 
are  executing  events  to  help  ensure  consistency  across  the  organization,  which  is  a  key 
document  strength. 
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Figure  5.  SETR  Tailoring  In  Relation  to  Acquisition  Life  Cycle.  Source:  DON 

MCSC  (2014,  3). 

SPAWARINST  5401.4,  which  is  titled  Systems  Engineering  Policy,  establishes 
the  systems  engineering  policy  for  the  command.  Within  this  instruction,  it  directs  that 
SPAWAR  5.0  will  conduct  SETRs  to  support  “the  delivery  of  an  integrated, 
interoperable,  and  tested  capability  to  the  Fleet”  (DON  SPAWAR  2016b,  2).  As  stated  in 
the  SETR  overview,  this  will  be  documented  by  programs  in  the  SEP;  but  in  addition 
SPAWAR  mandates  that  it  must  be  completed  and  submitted  to  SPAWAR  5.0  within 
three  months  of  program  initiation.  From  a  SPAWAR  perspective,  this  is  the  first 
instruction  that  includes  guidance  on  the  SETR  implementation. 

SPAWARINST  5400. 3A,  which  is  titled  Systems  Engineering  Technical  Review 
Policy,  establishes  the  policy  and  responsibilities  associated  with  the  SETR 
implementation  for  SPAWAR.  Key  roles  are  held  by  the  SPAWAR  CHENG  and  the 
respective  PM  when  conducting  the  SETR.  The  associated  enclosure,  which  is  the 
Systems  Engineering  Technical  Review  Organization  Standard  Process  Elandbook, 
provides  the  core  of  the  implementation  guidelines  to  the  SYSCOM. 

SPAWAR’ s  Systems  Engineering  Technical  Review  Organization  Standard 
Process  Handbook  goes  into  the  most  detailed  tailoring  options  of  all  the  SYSCOM 
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documentation.  The  handbook  provides  a  SETR  implementation  for  each  of  the  following 
models  that  can  be  implemented  within  the  SYSCOM: 

•  Model  1 :  Hardware  Intensive  Program  (traditional  SETR) 

•  Model  2:  Defense  Unique  Software  Intensive  Program 

•  Model  3:  Incrementally  Fielded  Software  Intensive  Program 

•  Model  4:  Accelerated  Acquisition  Program 

Model  1  aligns  to  what  was  discussed  in  the  SETR  overview.  However,  Figure  6  shows 
an  image  to  keep  context  as  each  of  the  additional  models  are  reviewed. 
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Figure  6.  Model  1  Hardware  Intensive  Program.  Source:  DON  SPAWAR 

(2016c,  18). 


Model  2  comes  into  play  when  there  is  a  defense  unique  software  intensive  program.  The 
hardware  has  to  be  capable  of  being  separated  from  the  software  builds  to  allow  more 
tailored  approach  on  the  SETR  side  for  software.  However,  the  combined  software  and 
hardware  builds  are  deployed  on  a  consistent  combined  schedule.  This  approach  would 
be  used  when  several  software  builds  are  necessary  for  a  hardware  build  that  is  being 
deployed.  This  is  the  first  introduction  for  SPAWAR  of  the  Build  Technical  Review 
(BTR)  and  Fielding  Technical  Review  (FTR)  events,  which  again  would  be  used  for  the 
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software  while  the  hardware  follows  a  more  typical  SETR  implementation  (DON 
SPAWAR  2016c).  These  two  new  reviews  are  designed  to  “assess  systems  engineering 
rigor  for  requirements  decomposition,  design,  development,  and  testing  to  allow  greater 
flexibility  and  response  to  evolving  technologies”  per  the  handbook  (22).  In  this  case,  the 
BTR  would  be  supporting  the  incremental  build  and  the  FTR  would  be  supporting  the 
software  side  of  the  combined  hardware/software  fielding  decision  that  a  program  would 
have  for  a  traditional  SETR  event.  Figure  7  shows  this  in  an  image  to  get  a  better  idea  of 
how  the  risk  reduction  occurs  from  a  software  perspective  with  the  multiple  BTR  events 
during  a  normal  timeframe  to  gain  a  better  understanding  of  the  technical  maturity  of  the 
defense  software  and  readiness  to  move  to  the  next  stage  of  the  program’s  life  cycle. 
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Figure  7.  Model  2  Defense  Unique  Software  Intensive  Program. 

Source:  DON  SPAWAR  (2016c,  19). 


The  next  model  in  the  SPAWAR  Systems  Engineering  Technical  Review  Organization 
Standard  Process  Handbook  takes  the  biggest  departure  from  the  traditional  SETR 
implementation  with  incrementally  fielded  software  intensive  program  model.  Model  3 
also  makes  use  of  the  BTR  and  FTR  concepts.  Due  to  the  more  pure  software  nature  of 
the  program  coupled  with  the  new  SETR  events  instead  of  the  traditional  SETR  events, 
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the  program  can  make  more  rapid  software  deployments  to  the  warfighter  as  shown  in 
Figure  8  by  the  incremental  builds  depicted  in  each  SETR  “row.”  The  SW  would  be  built 
agnostic  to  the  specific  HW,  but  compliant  with  HW  standards. 
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Figure  8.  Model  3  Incrementally  Fielded  Software  Intensive  Program. 

Source:  DON  SPAWAR  (2016c,  23). 


The  final  model  shown  in  Figure  9  and  provided  by  the  SPAWAR  Systems  Engineering 
Technical  Review  Organization  Standard  Process  Handbook  is  the  most  rapid  SETR 
model  depicted.  The  handbook  points  out  that  this  model  should  be  used  when  “schedule 
dominates  over  cost  and  technical  risk  considerations”  and  should  be  used  when 
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“technological  surprises  by  a  potential  adversary  necessitates  a  higher-risk  acquisition 
program”  (24).  So  even  though  it  is  the  most  rapid,  it  is  rather  specific  in  terms  of  when  it 
should  be  leveraged.  This  model  introduces  an  additional  event  in  the  handbook  called 
the  SETR-Lite  Engineering  Review  (SER),  which,  as  it  sounds,  is  a  highly  tailored  and 
focused  review  to  support  a  specific  program  decision;  but  still  contains  the  rigor  of  other 
SETR  events.  In  an  accelerated  acquisition  program  as  depicted  in  Figure  9,  a  SER  is 
executed  at  a  minimum  twice.  However,  the  SER  is  a  useful  review  for  projects  that  may 
not  meet  the  threshold  for  a  traditional  or  even  tailored  SETR  implementation,  so  a  SER 
can  be  executed  to  meet  the  intent  of  an  independent  technical  review  as  often  as  needed 
in  the  project  life  cycle. 
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Figure  9.  Model  4  Accelerated  Acquisition  Program. 

Source:  DON  SPAWAR  (2016c,  25). 


E.  INDUSTRY 


IEEE  Standard  15288.2-2014,  which  is  the  Standard  for  Technical  Reviews  and 
Audits  on  Defense  Programs,  details  the  traditional  SETR  implementation;  however, 
much  like  described  in  the  process  tailoring  section  for  IEEE  Standard  15288,  there  is  a 
section  up  front  in  the  document  that  speaks  to  process  tailoring  and  expectations.  Unlike 
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the  DOD  or  DON  documentation,  the  IEEE  Standard  15288.2-2014  includes  a  solid 
baseline  definition  list,  which  includes  items  such  as  the  following: 

acceptability  criteria:  A  documented  set  of  characteristics  of  a  program’s 
work  products  that  if  satisfied,  forms  a  sufficient  basis  for  judging  each 
product’s  content  to  be  acceptable  to  support  a  successful  review  or  audit. 

entry  criteria:  Artifacts  and  other  review  or  audit  elements  that  must  be 
completed  before  the  review  or  audit  can  be  conducted. 

exit  criteria:  Review  or  audit  elements  that  must  be  assessed,  completed, 
and  action  items  closed  before  successful  completion  of  the  technical 
review  or  audit  can  be  declared. 

technical  reviews:  A  series  of  systems  engineering  activities  conducted  at 
logical  transition  points  in  a  system  life  cycle,  by  which  the  progress  of  a 
program  is  assessed  relative  to  its  technical  requirements  using  a  mutually 
agreed-upon  set  of  criteria.  (IEEE  Computer  Society  2014,  4-5) 

The  IEEE  Standard  15288.2-2014  goes  through  each  traditional  SETR  event  to  explain 
what  makes  up  a  successful  event.  Starting  off,  each  section  explains  the  acceptability 
criteria  for  the  event  with  regards  to  each  product  (e.g.,  system  specification)  that  will  be 
reviewed  at  the  event.  The  standard  moves  on  to  detail  what  is  required  for  event 
preparation  broken  down  by  the  responsible  person  and  their  associated  actions.  The 
actual  execution  of  the  event  is  then  explained  along  with  what  will  be  reviewed  in  each 
part  of  the  SETR  event  itself.  Finally,  this  section  of  the  standard  speaks  to  what  is 
needed  to  successfully  close  out  this  SETR  event  from  each  person  and  for  what  actions 
are  they  responsible.  While  the  NAVSEA  05H  Technical  Review  Manual  contained  great 
detail  about  planning  to  closure  of  a  SETR  event,  this  IEEE  Standard  15288.2-2014  goes 
step  by  step  for  each  traditional  SETR  event  including  what  it  takes  from  planning  all  the 
way  to  closure  of  each  individual  event,  which  is  similar  to  the  level  of  detail  in  the 
MCSC  SETR  Handbook. 

The  Software  Engineering  Institute  (SEI)  released  a  technical  note  on  Agile 

Methods:  Selected  DOD  Management  and  Acquisition  Concerns  in  October  2011,  which 

frames  the  use  of  agile  as  a  mechanism  to  address  the  “disconnect  between  the 

warfighter/demand  tempo  and  the  acquisition/contracting  tempo”  (Software  Engineering 

Institute  [SEI]  2011,  3).  The  technical  note  addresses  topics  such  as  contract  execution, 
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contract  monitoring,  and  sustainment  when  a  program  is  leveraging  agile  development 
methodology.  Of  note  from  a  SETR  perspective,  the  technical  note  describes  two 
potential  solutions  for  dealing  with  more  traditional  technical  reviews  to  include  specific 
advantages  and  disadvantages  of  each.  The  first  option  is  translation  of  agile  products 
into  the  traditional  review  events.  Whether  this  translation  is  done  by  the  vendor 
themselves  or  another  support  entity  (e.g.,  Systems  Engineering  and  Technical  Assistance 
contractor),  the  program,  as  pointed  out  by  the  SEI  technical  note,  is  incurring  a  cost  to 
translate  the  agile  material  into  traditional  documents,  which  may  outweigh  the  benefit 
agile  provides.  The  second  option  the  SEI  technical  note  provides  is  to  execute  more 
iterative  reviews,  or  mini  events,  that  build  up  to  a  more  traditional  SETR  event.  This 
second  option  is  similar  to  the  tailored  NAV AIR’s  RBR  or  SPAWAR’s  BTR. 

F.  SUMMARY 

The  DOD  and  DON  directives  and  instructions  lay  the  system  engineering 
foundation  for  the  naval  SYSCOMs  guidance  regarding  SETR  implementation.  The 
following  are  the  key  attributes  from  each  of  the  higher  level  DOD  and  DON  documents 
reviewed  in  this  chapter: 

•  The  Defense  Acquisition  System  (DODD  5000.01):  Lays  the  foundation 
with  a  standard  definition  of  what  systems  engineering  is  within  the  DOD. 

•  Operations  of  the  Defense  Acquisition  System  (DODI  5000.02):  Includes 
requirement  for  PDR  and  CDR  within  DOD  level  instruction  along  with 
criteria  for  when  applicable. 

•  Business  Systems  Requirements  and  Acquisition  (DODI  5000.75): 
Supersedes  DODI  5000.02  for  defense  business  systems;  includes 
reference  to  technical  assessments  that  must  be  addressed  in  the 
implementation  plan. 

•  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  (DODI 
5134.16):  Initial  SEP  requirement  introduced  at  DOD  level. 

•  ASN  RD&A  Memo,  SETR  Process  for  Naval  Acquisition  Programs,  13 
June  08:  Directs  SETR  implementation  across  naval  SYSCOMs. 

•  Naval  SYSCOM  Systems  Engineering  Policy  (SPAWARINST  5000.1): 
Specifies  the  minimum  SETR  elements  (e.g.,  SETR  chair,  agenda, 
schedule,  expected  results)  across  the  naval  SYSCOMs. 
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•  DON  Implementation  and  Operation  of  the  Defense  Acquisition  System 
and  the  Joint  Capabilities  Integration  and  Development  System 
(SECNAVINST  5000. 2E):  Introduction  of  “IT  Box”  concept.  Underscores 
the  importance  of  being  able  to  tailor  the  program’s  SETR  implementation 
to  keep  pace  with  the  increased  design  to  deployment  timeline  that  “IT 
Box”  enables. 

Each  SYSCOM  has  varying  guidance  regarding  SETR  to  include  tailoring.  The 
following  are  the  key  attributes  from  each  of  the  SYSCOM  documents  reviewed  in  this 
chapter: 

•  NAVSEA  05H  Technical  Review  Manual,  Version  2.0,  18  December 
2009:  Provides  most  comprehensive  traditional  SETR  guidance,  but  does 
have  reminders  for  tailoring  mentioned  throughout.  Appendix  includes 
detailed  descriptions  of  entrance  and  exit  criteria  for  each  SETR  event, 
different  documentation,  architectural  views,  and  scheduling  details. 

•  NAVAIR  Systems  Engineering  Technical  Review  Process  (NAVAIRINST 

4355. 19E):  Includes  specific  definition  for  tailoring  SETR 

implementation.  Introduces  a  tailored  event — RBR — for  agile  software 
development. 

•  NAVAIR  Adapting  Acquisition  to  Agile  Software  Development:  A  How- 
To  Guide,  Version  2.0,  19  March  2014:  Provides  execution  level  details  on 
the  RBR. 

•  MCSC  SETR  Handbook  (SIAT-HDBK-001):  Encourages  tailoring 
opportunities  and  even  introduces  the  NIR,  which  is  a  tailored  event. 
Detailed  appendix  for  each  SETR  event  that  includes  an  event  overview, 
entrance  criteria,  review  elements,  agenda,  exit  criteria,  and  evaluation 
criteria 

•  SPAWAR  Systems  Engineering  Policy  (SPAWARINST  5401.4): 
Mandates  SEP,  which  is  where  SETR  implementation  is  documented, 
must  be  completed  and  submitted  to  SPAWAR  5.0  within  three  months  of 
program  initiation. 

•  SPAWAR  Systems  Engineering  Technical  Review  Policy  (SPAWARINST 
5400. 3A):  Establishes  key  SETR  roles  within  SPAWAR. 

•  SPAWAR  Systems  Engineering  Technical  Review  Organizational 
Standard  Process  Handbook :  Provides  a  tailored  SETR  implementation 
for  each  DOD  software  acquisition  models,  which  is  the  most  extensive 
tailoring  guidance  in  all  of  the  SYSCOM  documents. 
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The  following  list  includes  key  attributes  from  the  industry  documents  reviewed 
in  this  chapter: 

•  IEEE  Technical  Reviews  and  Audits  for  Defense  standard:  provides  solid 
definitions  (e.g.,  exit  criteria,  entrance  criteria).  With  each  event,  the 
standard  provides  preparation  instruction  and  other  information  to  ensure 
the  event  is  successful. 

•  SEI  Agile  Methods:  Selected  DOD  Management  and  Acquisition 
Concerns :  addresses  topics  such  as  contract  execution,  contract 
monitoring,  and  sustainment  when  a  program  is  leveraging  agile 
development  methodology.  Describes  two  potential  solutions  for  dealing 
with  more  traditional  technical  reviews 

The  following  chapter  presents  details  on  the  method  for  collecting  the  data  as 
well  as  presentation  of  the  data  gathered  from  the  survey. 


33 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


34 


IV.  PRESENTATION  OF  SURVEY  DATA 


A.  INTRODUCTION 

To  address  the  research  questions,  a  survey  was  leveraged  to  gather  data  points 
from  SMEs  executing  SETRs  to  understand  how  SETRs  are  being  tailored,  what  impacts 
the  SETR  has  on  future  acquisition  steps,  and  what  hurdles  programs  are  currently  facing. 
These  survey  responses  are  in  the  context  of  the  current  policies,  instructions,  and  other 
job  aides  that  are  in  place  across  the  DOD,  DON,  and  naval  SYSCOMs.  Upon  conclusion 
of  the  analysis,  recommendations  will  be  provided  based  on  lessons  learned  on  how  to 
increase  likelihood  of  a  program  ROI  for  a  tailored  SETR  implementation  as  it  relates  to 
naval  software  acquisitions.  This  chapter  will  provide  details  on  the  method  for  collecting 
the  data  as  well  as  presentation  of  the  survey  data  gathered.  Appendix  B  provides  the 
comprehensive  survey  data  set.  Chapter  V  provides  the  research  analysis  and  discussion 
tying  the  policy  review  chapter  to  the  data  from  the  survey  to  address  the  research 
questions. 

As  suggested  in  Fowler’s  Survey  Research  Methods,  one  of  the  first  steps  in 
initiating  a  survey  is  choosing  the  appropriate  sample  frame  to  understand  those  who  will 
be  selected  to  participate  in  the  survey  as  the  target  population  (Fowler  2014).  For  this 
specific  research,  the  survey  was  distributed  across  the  engineering  workforce  at 
SPAWAR  Systems  Center  Atlantic  (SSC  LANT)  as  well  as  shared  with  the  other  naval 
SYSCOMs  where  feasible.  The  target  audience  within  these  organizations  was  SMEs 
who  have  expertise  organizing,  leading,  and/or  participating  in  a  program  SETR  event. 
SSC  LANT  engineering  technical  workforce  is  made  up  of  about  3,000  government 
employees  (Figure  10).  The  survey  was  focused  within  this  engineering  technical 
workforce  to  a  more  granular  audience  of  SETR  SMEs.  SSC  LANT  supports  other  DON 
customers  (Figure  11)  as  a  working  capital  funded  organization,  which  provided  a  unique 
opportunity  to  gain  insight  into  other  DON  program  SETR  implementations  through  this 
survey  population.  The  top  five  sponsors  include  four  of  the  naval  SYSCOMs — 
SPAWAR,  MCSC,  NAVSEA,  and  NAVAIR.  As  the  SSC  LANT  technical  workforce 
supports  programs  across  these  other  SYSCOMs,  the  survey  generated  results  that 
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provided  insight  across  DON  programs  more  completely  than  a  survey  of  a  single 
SYSCOM. 
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Figure  10.  Command  Technical  Population.  Source:  Department  of  Navy  (DON) 
Space  and  Naval  Warfare  Systems  Command  Systems  Center  Atlantic 

(SSC  LANT)  (2017). 
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Figure  11.  Command  Workforce  Profile.  Source:  DON  SSC  LANT  (2017). 


The  online  SurveyMonkey  tool  was  used  to  distribute  and  collect  the  data  from  26 
April  through  17  May  2017.  Overall,  the  survey  had  48  respondents.  The  survey  did  not 
require  a  user  to  provide  identifying  information,  other  than  what  organization  they  most 
closely  associated  with  in  order  to  understand  the  breadth  of  organizations  that  were 
sampled  in  the  survey.  The  Naval  Postgraduate  School  Institutional  Review  Board 
determined  that  the  research  is  not  designed  to  collect  information  about  individuals  and 
therefore  is  not  human  subject  research.  The  survey  questions,  which  are  provided  in 
Table  4,  did  collect  a  mix  of  objective  and  subjective  data  points.  The  objective  data 
points  consisted  of  those  that  are  factually  based,  such  as  size  of  program,  events  tailored, 
and  area  of  program  adjusted  (Fowler  2014).  The  subjective  data  points  relate  to  a  SMEs 
experience  such  as  what  events  were  most  valuable.  These  questions  were  worded  to 
reduce  error  such  as  personalizing  the  response  by  ensuring  that  the  program  was  the 
acting  noun  not  the  person. 
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Table  4.  Survey  Questions  and  Potential  Responses 


No. 

Survey  Questions 

Question  Type 

Response  Options 

1 

Which  organization  are 
you  most  closely 
associated  with? 

Multiple  Choice 
(Single  Answer) 

Department  of  Defense 

Department  of  Navy 

Marine  Corps  Systems  Command 

Naval  Air  Systems  Command 

Naval  Facilities  Engineering  Command 

Naval  Sea  Systems  Command 

Naval  Supply  Systems  Command 

Space  and  Naval  Warfare  Systems  Command 
Other 

2 

What  is  the  size  of  the 
program  that  used  the 

SETR  process? 

Multiple  Choice 
(Single  Answer) 

ACATI 

ACAT  II 

ACAT  III 

Other  (please  specify) 

3 

What  type  of  software 
acquisition  model  did  this 
program  leverage? 

Multiple  Choice 
(Single  Answer) 

Commercial  off-the-shelf 

Government  off-the-shelf 

New  Development 

Other  (please  specify) 

4 

Did  the  program  tailor  the 
system  engineering 
technical  review  process 
to  support  your  program? 

Multiple  Choice 
(Single  Answer) 

Yes 

No 

5 

If  applicable,  which  part 
did  you  tailor? 

Multiple  Choice 
(Multiple  Answer) 

Review  Order 

Reviews  Included 

Entrance/Exit  Criteria 

Not  Applicable 

Other  (please  specify) 

6 

What  reviews  did  the 
program  include  in  the 
SETR  process? 

Multiple  Choice 
(Multiple  Answer) 

Build  Technical  Review 

Capability  Build  Review — Initial 

Capability  Build  Review — Design 

Capability  Build  Review — Readiness 

Critical  Design  Review 

Fielding  Technical  Review 

Functional  Configuration  Audit 

Integration  Readiness  Review 

Operational  Test  Readiness  Review 

Physical  Configuration  Audit 

Preliminary  Design  Review 

Production  Readiness  Review 

Release  Backlog  Review 

SETR-Lite  Engineering  Review 

Software  Specification  Review 

System  Functional  Review 

System  Requirements  Review 

System  Verification  Review 

Test  Readiness  Review 

Other  (please  specify) 
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No. 

Survey  Questions 

Question  Type 

Response  Options 

7 

Which  review  events  did 
the  program  find  most 
valuable  to  assessing  the 
technical  maturity  of  the 
program? 

Multiple  Choice 
(Multiple  Answer) 

Build  Technical  Review 

Capability  Build  Review — Initial 

Capability  Build  Review — Design 

Capability  Build  Review — Readiness 

Critical  Design  Review 

Fielding  Technical  Review 

Functional  Configuration  Audit 

Integration  Readiness  Review 

Operational  Test  Readiness  Review 

Physical  Configuration  Audit 

Preliminary  Design  Review 

Production  Readiness  Review 

Release  Backlog  Review 

SETR-Lite  Engineering  Review 

Software  Specification  Review 

System  Functional  Review 

System  Requirements  Review 

System  Verification  Review 

Test  Readiness  Review 

Other  (please  specify) 

8 

What  area  of  the  program 
was  adjusted  based  on  the 
outcome  of  the  SETR? 

Multiple  Choice 
(Multiple  Answer) 

Cost 

Schedule 

Technical  Design 

No  change 

Other  (please  specify) 

9 

By  what  percentage  of  the 
cost,  schedule,  and/or 
technical  design  was  the 
program  adjusted,  if 
applicable? 

Multiple  Choice 
(Single  Answer) 

0-20% 

21-40% 

41-60% 

61-80% 

81-100% 

Not  Applicable 

10 

Did  any  of  the  following 
options  limit  or  create 
obstacles  if  the  program 
tailored  the  SETR 
implementation? 

Multiple  Choice 
(Multiple  Answer) 

Internal  Organization 

External  Organization 

Policy 

No  limitation 

Not  applicable 

Other  (please  specify) 

11 

Did  the  program  include 
any  metrics  to  assess  the 
return  on  investment  from 
the  reviews? 

Multiple  Choice 
(Single  Answer) 

Yes  (please  specify).  No 

12 

Did  the  program  have 
consistent  leadership 
throughout  a  complete 
SETR  cycle? 

Multiple  Choice 
(Single  Answer) 

Yes,  No 

13 

Please  provide  any 
additional  feedback 
regarding  SETR 
implementations. 

Free  Text 

N/A 
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B.  SURVEY  DATA 


(1)  Question  1 

Figure  12  provides  the  results  to  the  initial  question — ’’Which  organization  are 
you  most  closely  associated  with?”  The  intent  of  this  question  was  to  understand  from  a 
sampling  perspective  which  organizations  were  covered  or  not  during  the  survey  period. 
SPAWAR  was  the  SYSCOM  that  the  majority  of  participants  most  closely  associated 
themselves  with,  followed  by  the  DON. 


Q1  Which  organization  are  you  most  closely 
associated  with? 

Answered:  48  Skipped:  0 
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Figure  12.  Question  1  Survey  Results 
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(2)  Question  2 

Figure  13  provides  the  results  to  the  second  question — ’’What  is  the  size  of  the 
program  that  used  the  SETR  process?”  This  question  was  included  to  understand,  from  a 
sampling  perspective  what  mixture  of  AC  AT  program  types  was  covered  in  the  survey. 
Most  survey  participants  were  involved  in  ACAT  I  programs.  Seven  of  the  nineteen 
responses  in  the  “other”  category  specified  that  the  program  size  was  an  abbreviated 
acquisition  program  (AAP).  Per  SECNAVINST  5000. 2E,  AAP  are  programs  at  a  high 
level  that  do  not  require  operational  test  and  evaluation  as  documented  by  the  Operational 
Testing  Agency  (OTA)  and  meet  a  specific  dollar  threshold  per  the  instruction.  The 
complete  set  of  responses  provided  in  the  free  text  field — ’’Other  (please  specify)” — are 
included  in  Appendix  B . 


What  is  the  size  of  the  program  that  used 
the  SETR  process? 

Answered:  48  Skipped:  0 
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Figure  13.  Question  2  Survey  Results 
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(3)  Question  3 

Figure  14  provides  the  results  to  the  next  question — ’’What  type  of  software 
acquisition  model  did  this  program  leverage?”  This  question  was  incorporated  into  the 
survey  to  help  answer  the  research  question  regarding  if  there  were  any  specific  tailoring 
concerns  based  on  the  type  of  acquisition  model  the  program  leveraged.  Most  of  the 
participants  acquired  COTS  based  on  the  responses  provided.  Ten  of  the  thirteen  “other” 
responses  included  some  combination  of  the  original  response  options  provided  (e.g., 
COTS  and  GOTS,  all).  The  complete  set  of  responses  provided  in  the  free  text  field — 
’’Other  (please  specify)” — are  included  in  Appendix  B. 


Q3  What  type  of  software  acquisition  model 
did  this  program  leverage? 

Answered:  48  Skipped:  0 
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Figure  14.  Question  3  Survey  Results 
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(4)  Question  4 

Figure  15  provides  the  results  to  the  fourth  question — ’’Did  the  program  tailor  the 
SETR  process  to  support  your  program?”  This  question  begins  to  get  at  the  heart  of  the 
thesis  regarding  whether  programs  are  leveraging  the  tailoring  allowed  by  the  current 
policies  in  places  across  the  DOD,  DON,  and  their  respective  SYSCOMs.  Based  on  the 
participant’s  responses,  programs  do  appear  to  be  taking  advantage  of  the  various 
tailoring  opportunities  described  in  the  current  policies  as  over  80%  responded  yes  the 
program  did  tailor  the  SETR  implementation. 


Q4  Did  the  program  tailor  the  system 
engineering  technical  review  process? 

An+wertd:  48  SklppeO;  C 
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Figure  15.  Question  4  Survey  Results 

(5)  Question  5 

Figure  16  provides  the  results  to  the  fifth  question,  which  is  a  follow  on  to  the 
previous  question — ”If  applicable,  which  part  did  you  tailor?”  The  intent  behind  this 
question  is  to  understand  how  the  programs  tailored  by  using  this  multi-select  question.  A 
program  could  respond  with  not  applicable  or  multiple  other  options.  Tailoring  reviews 
included  and  entrance/exit  criteria  were  the  most  common  responses  from  survey 
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participants.  Of  the  eight  “other”  responses,  three  of  them  included  skipping  required 
documentation.  The  complete  set  of  responses  provided  in  the  free  text  field — ’’Other 
(please  specify)” — are  included  in  Appendix  B. 


If  applicable,  which  part  did  you  tailor? 

Answered:  44  Skipped:  4 
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Figure  16.  Question  5  Survey  Results 


(6)  Question  6 

Figure  17  provides  the  results  to  the  next  question — ’’What  reviews  did  the 
program  include  in  the  SETR  process?”  This  question  was  to  baseline  which  reviews 
from  the  potential  options  provided  in  the  SYSCOM  policy  documentation  to  include  the 
known  tailored  reviews  (e.g.,  NAVAIR  RBR,  MCSC  NIR,  SPAWAR  SER).  The  most 
commonly  included  reviews  were  PDR,  CDR,  SRR,  and  TRR.  The  specific  responses  in 
the  “other”  category  listed  tailored  events  such  as  combining  SRR  and  PDR  into  a  single 
event,  so  there  was  not  a  consistently  repeated  answer.  The  complete  set  of  responses 
provided  in  the  free  text  field — ’’Other  (please  specify)” — are  included  in  Appendix  B. 


44 


Q6  What  reviews  did  the  program  include  in 
the  SETR  process? 
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Figure  17.  Question  6  Survey  Results 
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(7)  Question  7 

Figure  18  provides  the  results  to  the  seventh  question — ’’Which  review  events  did 
the  program  find  most  valuable  to  assessing  the  technical  maturity  of  the  program?”  This 
question  was  included  to  address  the  research  question  focused  on  the  critical  reviews  of 
the  SETR  implementation  that  program  managers  and/or  lead  engineers  should  include  in 
a  tailored  implementation.  The  most  common  answer  to  this  question  was  PDR,  CDR, 
and  TRR,  which  was  closely  followed  by  SRR.  The  complete  set  of  responses  provided 
in  the  free  text  field — ’’Other  (please  specify)” — are  included  in  Appendix  B. 


46 


Q  J  Which  review  event*  did  I  he  program 
find  most  valuable  to  asse^ing  the 
technical  maturity  of  the  program? 
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Figure  18.  Question  7  Survey  Results 
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(8)  Question  8 

Figure  19  provides  the  results  to  the  next  question — ’’What  area  of  the  program 
was  adjusted  based  on  the  outcome  of  the  SETR?”  The  outcome  of  a  SETR 
implementation  overall  should  be  to  ensure  that  the  program  is  on  track,  or  adjust  based 
on  the  risk  found  in  the  review  to  ensure  follow  on  phases  of  the  acquisition  process  do 
not  increase  in  risk.  This  question  was  intended  to  understand  whether  based  on  the 
SETR  events,  the  program  had  to  adjust  the  cost,  schedule,  technical  design,  or  some 
other  part  of  the  program  prior  to  proceeding  to  the  next  acquisition  phase.  Schedule  and 
technical  design  were  the  top  two  responses  indicating  areas  the  program  adjusted  based 
on  the  outcome  of  the  SETR.  The  complete  set  of  responses  provided  in  the  free  text 
field — ’’Other  (please  specify)” — are  included  in  Appendix  B. 


Q8  What  area  of  the  program  was  adjusted 
based  on  the  outcome  of  the  SETR? 

Answered:  48  Skipped:  0 


Figure  19.  Question  8  Survey  Results 
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(9)  Question  9 

Figure  20  provides  the  results  to  the  ninth  question — ”By  what  percentage  of  the 
cost,  schedule,  and/or  technical  design  was  the  program  adjusted,  if  applicable?”  If  the 
answer  to  the  previous  question  indicated  there  was  an  adjustment  made,  then  this 
question  sought  to  understand  how  significantly  the  program  adjusted  after  the  SETR 
review.  If  an  adjustment  was  made  to  the  program  based  on  the  SETR  outcome,  based  on 
the  responses  to  this  survey,  the  change  would  most  likely  be  in  the  0-20%  range,  which 
was  the  response  for  over  half  the  participants. 


Q9  By  what  percentage  of  the  cost, 
schedule,  and/or  technical  design  was  the 
program  adjusted,  if  applicable? 
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Figure  20.  Question  9  Survey  Results 
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(10)  Question  10 

Figure  21  provides  the  results  to  the  next  question — ’’Did  any  of  the  following 
options  limit  or  create  obstacles  if  the  program  tailored  the  SETR  implementation?”  This 
survey  question  aligns  to  a  research  question  focused  on  addressing  the  obstacles  (e.g., 
policy,  timelines,  and  maturity)  and  risks  faced  by  naval  programs  that  have  attempted  to 
tailor  the  SETR  process.  More  specifically  do  these  obstacles  vary  based  on  the  size  of 
the  program  (e.g.,  ACAT  I  vs  non  program  of  record)?  The  most  common  response  to 
this  survey  question  was  that  the  external  organization  created  limits  and/or  obstacles  to 
tailoring  the  SETR  implementation.  The  complete  set  of  responses  provided  in  the  free 
text  field — ’’Other  (please  specify)” — are  included  in  Appendix  B. 


Q10  Did  any  of  the  following  options  limit  or 
create  obstacles  if  the  program  tailored  the 
SETR  implementation? 
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Figure  21.  Question  10  Survey  Results 
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(11)  Question  11 

Figure  22  provides  the  results  to  the  eleventh  question — ’’Did  the  program  include 
any  metrics  to  assess  the  return  on  investment  from  the  reviews?”  One  area  that  was  not 
addressed  heavily  in  any  of  the  government  policy  documentation  was  metrics,  so  this 
question  was  included  to  see  if  lack  of  mandatory  inclusion  carried  through  to  the  reviews 
themselves.  The  SPAWAR  Systems  Engineering  Technical  Review  Organization 
Standard  Process  Handbook  did  include  some  specific  metric  examples  as  noted  in  Table 
2.  SETR  implementations  are  iterative  events  throughout  different  phases  of  the  program 
life  cycle  as  mentioned  at  various  points  in  this  thesis.  Metrics  are  one  way  to  baseline 
the  outcome  of  an  event  and,  as  follow  on  events  occur,  understand  if  improvements  are 
occurring.  Based  on  the  responses  to  the  survey,  the  policy  seems  to  be  indicative  of  the 
actual  execution  of  the  SETR  events  as  almost  80%  did  not  include  any  metrics  to 
understand  if  there  was  a  ROI  from  the  reviews.  The  complete  set  of  responses  provided 
in  the  free  text  field — ”Yes  (please  specify)” — are  included  in  Appendix  B. 


Q11  Did  the  program  include  any  metrics  to 
assess  the  return  on  investment  from  the 

reviews? 

Answered:  47  Skipped:  1 
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Figure  22.  Question  11  Survey  Results 
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(12)  Question  12 

Figure  23  provides  the  results  to  the  twelfth  question — ’’Did  the  program  have 
consistent  leadership  throughout  a  complete  SETR  cycle?”  Leadership  as  highlighted 
throughout  each  of  the  DOD,  DON,  and  SYSCOM  documentation  primarily  through  the 
program  PM  is  critical  to  ensuring  successful  SETR  implementation,  so  this  question  was 
included  to  see  if  the  programs  included  in  the  survey  had  consistent  leadership  through  a 
complete  SETR  cycle.  The  response  from  this  question  appears  fairly  split  with  almost 
half  having  consistent  leadership  throughout  a  complete  SETR  cycle  and  the  other  not 
having  consistent  leadership. 


Q12  Did  the  program  have  consistent 
leadership  throughout  a  complete  SETR 

cycle? 

Answered:  47  Skipped:  1 
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Figure  23.  Question  12  Survey  Results 
(13)  Question  13 

The  final  question  was  an  open  ended  question — ’’Please  provide  any  additional 
feedback  regarding  SETR  implementations.”  The  responses  varied  from  general  feedback 
regarding  SETR  to  how  tailoring  was  implemented  and  included  feedback  on  the  survey 
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itself.  The  complete  set  of  responses  provided  in  this  free  text  field  are  included  in 
Appendix  B.  This  chapter  provided  details  on  the  method  for  collecting  the  data  as  well 
as  the  data  gathered  from  the  survey.  Chapter  V  will  address  the  research  questions 
through  survey  data  analysis  and  details  from  the  policy  and  standard  review  to  include 
recommendations  based  on  the  responses  from  the  final  open  ended  question. 

C.  SUMMARY 

The  survey  was  distributed  across  the  engineering  workforce  SSC  LANT  as  well 
as  shared  with  the  naval  SYSCOMs  where  feasible.  The  target  audience  within  these 
organizations  was  the  SMEs  who  have  expertise  organizing,  leading,  and/or  participating 
in  a  SETR  event  for  a  program.  The  online  SurveyMonkey  tool  was  used  to  distribute  and 
collect  the  data  from  26  April  through  17  May  2017.  The  survey  did  not  require  a  user  to 
provide  identifying  information  other  than  what  organization  they  most  closely  associated 
with,  in  order  to  understand  the  breadth  of  SYSCOMs  that  were  sampled  in  the  survey. 
The  survey  questions  did  collect  a  mix  of  objective  and  subjective  data  points.  Overall, 
the  survey  had  48  respondents,  which  included  varying  ACAT  program  sizes,  software 
acquisition  models,  and  degrees  of  tailoring  the  process.  The  following  chapter  provides 
the  research  analysis,  which  ties  the  literature  review  section  to  the  data  from  the  survey 
and  addresses  the  research  questions. 
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V.  RESEARCH  ANALYSIS  AND  DISCUSSION 


A.  INTRODUCTION 


This  chapter  provides  the  research  analysis,  which  ties  the  literature  review 
chapter  to  the  survey  data  to  address  the  research  questions.  The  literature  review  focused 
on  the  DOD,  DON,  naval  SYSCOM  policies,  and  industry  documentation  providing 
direction  and  guidance  related  to  the  process,  entrance  criteria,  exit  criteria,  leadership 
involvement,  tailoring,  and  all  other  applicable  aspect  of  the  SETR  process.  Leveraging  a 
tailored  SETR  implementation  provides  the  necessary  structured  engineering  framework 
while  keeping  up  with  a  dynamic  software  development  environment  to  meet  increasing 
need  for  enhanced  capability  delivered  to  the  warfighter  in  a  shorter  timeframe. 
Specifically,  this  research  addresses  how  the  SETR  implementation  can  be  tailored  to 
most  efficiently  leverage  resources  and  minimize  schedule  impact  while  addressing  the 
acquisition,  technical,  and  policy/legal  requirements  within  naval  software  acquisitions. 
In  order  to  answer  this  question,  the  following  subsidiary  questions  will  also  be 
addressed. 


1.  What  are  the  obstacles  (e.g.,  policy,  timelines,  and  maturity)  and  risks  faced 
by  naval  programs  that  have  attempted  to  tailor  the  SETR  process?  Do  these 
obstacles  vary  based  on  the  size  of  the  program  (e.g.,  ACAT  I  vs  non 
program  of  record)? 

2.  What  are  the  critical  elements  of  the  current  system  engineering  technical 
review  process  (e.g.,  entrance/exit  criteria,  order  of  reviews,  risk 
assessment)  that  program  managers  and/or  lead  engineers  should  include  in 
a  tailored  implementation?  What  incentives  or  return  on  investment  do  these 
critical  elements  provide? 

3.  Are  there  any  considerations/differences  that  should  be  accounted  for  when 
tailoring  the  process  to  support  various  acquisition  models  (e.g.,  COTS,  new 
development,  GOTS)? 

4.  What  can  engineers  and  program  managers  of  future  naval  acquisitions 
learn  from  other  programs  that  have  attempted  to  tailor? 
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B.  RESEARCH  QUESTION  ANALYSIS 
1.  Primary  Research  Question 

The  primary  research  question  is  “how  can  the  SETR  implementation  be  tailored 
to  most  efficiently  leverage  resources  and  minimize  schedule  impact  while  addressing  the 
acquisition,  technical,  and  policy/legal  requirements  within  naval  software  acquisitions?” 
Frank  Kendall,  then  Acting  Undersecretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (AT&L),  addressed  the  question  of  what  is  the  optimal  program  structure, 
which  parallels  to  the  optimal  SETR  structure.  The  optimal  SETR  structure  is  dependent 
upon  many  variables  which,  IEEE  15288  Annex  A  provides  a  good  reference  for 
consideration. 

The  answer  to  the  question  is  either:  (A)  There  is  none,  or  (B)  There  are  an 
infinite  number.  There  is  no  one  best  way  to  structure  a  program.  Every 
program  has  its  own  best  structure,  and  that  structure  is  dependent  on  all 
the  many  variables  that  contribute  to  program  success  or  failure.... There  is 
no  one  solution.  What  I’m  looking  for  fundamentally  is  the  evidence  that 
the  program’s  leaders  have  thought  carefully  about  all  of  the  factors  that 
I’ve  mentioned —  and  many  others.  I  look  for  that  evidence  in  the  nature 
of  the  product  the  program  is  acquiring  and  in  the  structure  the  program’s 
leaders  have  chosen  to  use.  The  thinking  (and  the  supporting  data)  that 
went  into  determining  that  specific  and  often  unique  structure  is  what  I 
expect  to  see  in  an  acquisition  strategy,  and  it  is  what  I  expect  our  leaders 
to  be  able  to  explain  when  they  present  their  program  plans.  (Kendall 
2012,23) 

Over  80%  of  the  survey  respondents  from  across  various  DON  organizations  indicated, 
as  shown  in  Figure  15,  that  tailoring  was  occurring  within  programs.  In  a  tailored  SETR 
implementation,  the  programs  find  the  most  ROI  by  tailoring  entrance  criteria,  exit 
criteria,  and/or  which  specific  reviews  are  included  based  on  the  responses  to  survey 
question  five  as  shown  in  Figure  16.  Programs  are  primarily  adjusting  the  schedule  and/or 
technical  design  based  on  the  outcome  of  the  SETR  events  as  shown  in  Figure  18.  The 
percentage  of  adjustment  could  be  upwards  of  60%  based  on  the  response  to  question 
nine  as  shown  in  Figure  19.  Over  50%  of  the  survey  respondents  indicated  their  programs 
adjusted  the  cost,  schedule,  and/or  technical  design  by  up  to  20%.  However,  fifth 
question’s  response  structure  leads  to  some  uncertainty  in  this  result  as  the  answer 
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included  zero  as  part  of  the  option,  so  some  respondents  could  have  been  choosing  this 
option — ”0-20%” — to  indicate  no  change.  When  the  response  to  question  8,  regarding  the 
program  areas  that  were  adjusted  based  on  SETR  outcome,  are  compared  to  the 
percentage  of  adjustment  (question  9),  it  appears  that  four  of  the  responses  did  select  the 
option — ”0-20%” — to  indicate  no  change  as  shown  in  Table  5.  Table  6  provides  the 
original  response  count/percentages  and  the  revised  to  account  for  this  issue.  The  option 
“0-20%”  still  was  most  prevalent  response  with  the  adjustment  made  for  this  question 
response  structure  issue. 


Table  5.  Adjusted  program  areas  (Question  8)  Compared  to  Percent  Adjusted 

(Question  9) 


0-20% 

(1) 

21-40% 

(2) 

41-60% 

(3) 

61-80% 

(4) 

81  -1 00%  NOT  APPLICABLE 

(5)  (6) 

TOTAL 

-  Q8:  Cost  (A) 

78.57% 

11 

0.00% 

0 

21.43% 

3 

0.00% 

0 

0.00% 

0 

0.00% 

0 

30.43% 

14 

-  Q8: 

62.50% 

16.67% 

20.83% 

0.00% 

0.00% 

0.00% 

52.17% 

Schedule  (B) 

15 

4 

5 

0 

0 

0 

24 

-  Q8: 

68.00% 

8.00% 

16.00% 

0.00% 

0.00% 

8.00% 

54.35% 

Technical 

Design  (C) 

17 

2 

4 

0 

0 

2 

25 

-  Q8:  No 

36.36% 

0.00% 

0.00% 

0.00% 

0.00% 

63.64% 

23.91% 

change (D) 

4 

0 

0 

0 

0 

7 

11 

-  Total 

Respondents 

27 

4 

6 

0 

0 

9 

46 

Table  6.  Revised  Response  Calculations 


Question  9:  By  what  percentage  of  the  cost,  schedule,  and/or  technical  design  was 
the  program  adjusted,  if  applicable? 

Answer  Options 

Response 

Percent 

Response 

Count 

Revised 

Response 

Percent 

Revised 

Response 

Count 

0-20% 

56.3% 

27 

47.9% 

23 

21-40% 

8.3% 

4 

8.3% 

4 

41-60% 

12.5% 

6 

12.5% 

6 

61-80% 

0.0% 

0 

0.0% 

0 

81-100% 

0.0% 

0 

0.0% 

0 

Not  Applicable 

22.9% 

11 

31.3% 

15 

answered  question 

48 

skipped  question 

0 
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When  comparing  the  respondent’s  response  of  what  was  tailored  in  the  program 
(question  5)  to  the  percentage  the  program  cost,  schedule,  and/or  technical  design 
(question  9)  was  adjusted  as  shown  in  Table  7,  the  programs  that  tailored  the  SETR 
implementation  focusing  the  tailoring  on  which  reviews  to  include  seemed  to  have  the 
largest  return  based  on  the  percent  the  program  was  adjusted.  In  addition,  fewer  SETR 
events  should  indicate  less  time  and  resources  spent  preparing  for  unneeded  reviews. 
Additional  statistical  analysis  should  be  performed  in  this  area. 


Table  7.  SETR  Areas  Tailored  (Question  5)  Compared  to  Percent  Adjusted 

(Question  9) 


- 

0-20% 

(1) 

21-40% 

(2) 

41-60% 

(3) 

61-80% 

(4) 

81  -1 00%  NOT  APPLICABLE 

(5)  (6) 

TOTAL 

Q5:  Review 

16.67% 

16.67% 

33.33% 

0.00% 

0.00% 

33.33% 

13.95% 

Order  (A) 

1 

1 

2 

0 

0 

2 

6 

Q5:  Reviews 

54.84% 

9.68% 

19.35% 

0.00% 

0.00% 

16.13% 

72.09% 

Included  (B) 

17 

3 

6 

0 

0 

5 

31 

-  Q5: 

65.52% 

3.45% 

13.79% 

0.00% 

0.00% 

17.24% 

67.44% 

Entrance/Exit 
Criteria  (C) 

19 

1 

4 

0 

0 

5 

29 

-  Q5:  Not 

57.14% 

0.00% 

0.00% 

0.00% 

0.00% 

42.86% 

16.28% 

Applicable 

(D) 

4 

0 

0 

0 

0 

3 

7 

-  Total 

Respondents 

24 

3 

6 

0 

0 

10 

43 

Metrics  to  track  the  ROI  throughout  SETR  implementation  is  an  item  that  is 
difficult  to  address  based  on  respondent’s  answers  to  survey  question  eleven  as  shown  in 
Figure  22.  Only  nine  respondents  indicated  that  any  metrics  were  included  in  the  SETR 
process.  In  reviewing  the  additional  details  provided  in  the  open  ended  response  area  for 
this  question,  most  metrics  were  number  of  items  completed  (e.g.,  action  item  tracking). 
A  higher  level  of  action  items  completed  is  not  necessarily  a  metric  of  success  as  it  could 
indicate  that  the  program  should  not  have  entered  the  event  in  the  first  place  because  the 
program  was  not  sufficiently  ready.  Future  research  would  be  valuable  in  this  area  as  the 
policy  and  other  documentation  did  not  add  much  direction  and/or  examples.  From  a 
literature  review  perspective,  SPAWAR  Systems  Engineering  Technical  Review 
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Organization  Standard  Process  Handbook  was  the  only  document  that  addressed  metrics 
beyond  a  brief  mention. 

2.  Subsidiary  Question  1 

The  first  subsidiary  question  is  “what  are  the  obstacles  (e.g.,  policy,  timelines,  and 
maturity)  and  risks  faced  by  naval  programs  that  have  attempted  to  tailor  the  SETR 
process?  Do  these  obstacles  vary  based  on  the  size  of  the  program  (e.g.,  ACAT  I  vs  non 
program  of  record)?”  More  than  50%  of  the  survey  respondents  indicated  that  “External 
Organizations”  created  the  most  obstacles  (question  10)  as  shown  in  Figure  21. 
Interestingly,  when  reviewing  the  “other”  free  form  responses,  five  of  the  nine  responses 
could  have  also  been  categorized  as  “External  Organization”  as  shown  in  Table  8.  When 
looking  at  the  responses  to  question  10  through  the  lens  of  the  program  size  (e.g.,  ACAT 
I,  ACAT  II),  it  does  not  seem  to  affect  the  response  as  “External  Organization”  is 
consistently  the  response  more  than  50%  of  the  time  as  shown  in  Table  9.  Future  research 
should  include  statistical  analysis  in  this  area. 


Table  8.  Categorization  of  “Other”  Survey  Responses  for  Question  10 


Question  10:  Other  (please  specify) 

Categorization 

1 

It  depends.  For  instance,  if  the  PM/LSE  uses  a  SETR  event  that  senior 
leadership/MDA  is  not  familiar  with,  then  it  may  result  in  additional 
exchanges  between  the  PM  and  senior  leadership/MDA  to  educate  both 
sides  until  both  sides  agree  on  how  it  will  be  addressed  and  executed. 

External 

Organization 

2 

Every  Navy  ‘Heavy-weight’  that  controls  money  in  the  budget  thinks  they 
can  become  a  computer  guru  by  buying  the  latest  and  greatest  product  that 
a  salesperson  can  talk  them  into,  and  don’t  seem  to  realize  that  all 
requirements  of  ANY  systems  should  be  known  PRIOR  TO  any  purchase. 
Money  to  spend  doesn’t  make  a  manager  an  expert,  and  they  will  be 
rotated  out  of  that  job  in  (usually)  three  years  (or  less)  when  someone  else 
will  come  in  and  make  decisions  all  over  again.  If  you  don’t  know  how  a 
present  system  runs  (to  fulfill  all  the  laws,  policies,  required  today),  then 
how  can  you  say  that  you  are  an  expert  qualified  to  replace  that  system.  If 
you  force  the  conversion  on  partial  knowledge  then  the  risk  of  failure 
increases  and  all  members  of  that  service  suffer  and  Congress  is  unhappy. 
Why?  Because  they  controlled  the  budget  and  had  the  authority  but  did  not 
have  sufficient  knowledge,  or  training,  or  experience,  to  make  the  right  call 
knowing  the  risks. 

External 

Organization 

Personnel-  The  organization  was  not  staffed  with  the  appropriate  number 
of  people. 

Internal 

3 

Skillset  -The  work  force  was  not  trained  for  the  positions  they  held. 

Organization 
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Question  10:  Other  (please  specify) 

Categorization 

4 

PMW  240,  as  a  portfolio  of  smaller  programs,  is  built  around  the  idea  that 
the  SETR  process  needs  to  be  tailored  for  each,  individual  program; 
leadership  is  also  aware  that  defense  business  systems,  especially 
unmodified  COTS  implementations,  are  unique  among  DOD  acquisitions, 
which  necessitates  modifications  to  the  SETR  process 

Internal 

Organization 

5 

The  Program  Manager  did  not  and  currently  does  not  care  for  engineers. 
They  “get  in  his  way.” 

Maybe  that’s  why  his  program  is  failing...? 

Internal 

Organization 

6 

Actual  SETR  events  were  determined  by  MCSC  leadership  external  to  the 
Program  Office. 

External 

Organization 

7 

There  is  very  few  tailoring  guidelines.  This  leads  to  significant 
disagreement  about  what  tailoring  is  acceptable.  This  problem  exists  at  all 
levels.  Take  architecture  for  example:  the  DoDAF  specifications  are  vague 
on  tailoring,  the  Navy  Architecture  Development  Guide  does  not  directly 
address  tailoring,  and  the  SPAWAR  architecture  review  guidelines  (used  by 
contractors  to  conduct  reviews)  make  no  provision  for  tailoring.  Until 
tailoring  is  addressed  directly  at  all  levels  from  policy  to  review  guidelines, 
attempting  to  tailor  will  lead  to  significant  friction,  frustration,  and  delays. 

External 

Organization 

8 

The  external  organizations  (user  and  customer)  were  not  involved  in  the 
review.  Obstacles  arose  in  the  approval  of  a  design  that  the  user  had  never 
seen  and  making  design  decisions  driving  cost  and  schedule  that  the 
customer  did  not  have  input  to  change. 

External 

Organization 

Table  9.  Size  of  Program  (Question  2)  Compared  to  Obstacles  (Question  10) 


INTERNAL 
ORGANIZATION  - 
(1) 

EXTERNAL 
ORGANIZATION  - 
(2) 

POLICY 

(3) 

NO 

LIMITATION  - 
(4) 

NOT 

APPLICABLE  - 

(5) 

OTHER 

(PLEASE 

SPECIFY) 

(6) 

TOTAL 

-  02:  ACAT 1 

(A) 

29.41% 

5 

58.82% 

10 

17.65% 

3 

11.76% 

2 

11.76% 

2 

17.65% 

3 

Responses 

86.21% 

25 

-  Q2:  ACAT  II 

(B) 

0.00% 

0 

100.00% 

1 

0.00% 

0 

0.00% 

0 

0.00% 

0 

0.00% 

0 

Responses 

3.45% 

1 

_  Q2:  ACAT  III 

(C) 

18.18% 

2 

54.55% 

6 

0.00% 

0 

9.09% 

1 

18.18% 

2 

18.18% 

2 

Responses 

44.83% 

13 

Total 

7 

17 

3 

3 

4 

5 

29 

Respondents 


3.  Subsidiary  Question  2 

The  second  subsidiary  question  is  “what  are  the  critical  elements  of  the  current 
system  engineering  technical  review  process  (e.g.,  entrance/exit  criteria,  order  of  reviews, 
risk  assessment)  that  program  managers  and/or  lead  engineers  should  include  in  a  tailored 
implementation?  What  incentives  or  return  on  investment  do  these  critical  elements 
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provide?”  Based  on  the  survey  responses  when  a  program  tailors  its  SETR 
implementation,  the  primary  focus  is  on  the  reviews  included  and  what  entrance  and/or 
exit  criteria  is  included.  Interestingly,  in  the  open-ended  responses  to  survey  question  5 
the  respondents  agreed  that  tailoring  the  acquisition  documentation  was  another  key  area 
of  focus.  When  determining  which  events  were  most  valuable  (question  7)  to  include  in 
the  tailored  SETR  implementation,  PDR,  CDR,  and  TRR  ranked  20%  points  or  greater 
than  any  of  the  other  survey  options.  This  research  question  would  benefit  in  future 
research  from  additional  statistical  analysis  to  better  tie  the  return  on  investment  to  the 
type  of  tailoring  that  was  implemented. 

4.  Subsidiary  Question  3 

The  third  subsidiary  question  is  “are  there  any  considerations/differences  that 
should  be  accounted  for  when  tailoring  the  process  to  support  various  acquisition  models 
(e.g.,  COTS,  new  development,  GOTS)?”  Based  on  comparison  of  the  responses  in 
questions  3  and  4,  the  software  acquisition  model  does  not  affect  whether  the  program 
tailored  the  SETR  implementation  or  not  as  shown  in  Table  10.  Further,  Table  11 
indicates  that  the  software  acquisition  model  does  not  affect  a  program’s  focus  on  SETR 
tailoring  from  the  three  primary  elements:  the  reviews  included,  entrance  criteria,  and  exit 
criteria. 


Table  10.  Software  Acquisition  Model  (Question  3)  Compared  to  if  SETR 

Tailored  (Question  4) 

-  YES  (1)  -  NO  (2)  -  TOTAL 

-  Q3:  80.95%  19.05%  60.00% 

Commercial  17  4  21 

off  the  Shelf 

(A) 

-  Q3: 

Government 
off  the  Shelf 

(B) 

Q3:  New 
Development 

(C) 

-  Total  29  6  35 

Respondents 


75.00% 

3 


90.00% 

9 


25.00% 

1 


11.43% 

4 


10.00% 

1 


28.57% 

10 


61 


Table  11.  Software  Acquisition  Model  (Question  3)  Compared  to  SETR  Areas 

Tailored  (Question  5) 


REVIEW 
ORDER  (1) 

REVIEWS 

INCLUDED 

(2) 

ENTRANCE/EXIT 
CRITERIA  (3) 

NOT 

APPLICABLE  - 
(4) 

OTHER 
(PLEASE 
SPECIFY)  (5) 

TOTAL 

Q3: 

26.32% 

73.68% 

63.16% 

21.05% 

21.05% 

121.88% 

Commercial 
off  the  Shelf 
(A) 

5 

14 

12 

4 

4 

Responses 

39 

Q3: 

0.00% 

75.00% 

50.00% 

25.00% 

0.00% 

18.75% 

Government 
off  the  Shelf 
(B) 

0 

3 

2 

1 

0 

Responses 

6 

Q3:  New 

11.11% 

88.89% 

77.78% 

0.00% 

11.11% 

53.13% 

Development 

(C) 

1 

8 

7 

0 

1 

Responses 

17 

Total 

Respondents 

6 

25 

21 

5 

5 

32 

When  comparing  the  software  acquisition  model  (question  3)  to  the  specific 


reviews  included  (question  6)  as  shown  in  Table  12,  the  COTS  and  new  development 
models  included  most  consistently  the  PDR  and  CDR  as  SETR  events,  which  is  not 
surprising  these  are  the  two  reviews  given  DODI  5000.02  includes  the  requirement  for 
these  two  reviews  along  with  specific  criteria  on  when  the  requirement  applies. 
Expectations  for  these  two  events  are  better  understood  and  engrained  in  the  culture  due 
to  this  guidance  being  at  the  DOD  level.  The  comparison  data  on  the  GOTS  software 
acquisition  model  is  more  evenly  distributed  across  the  review  types  as  only  four 
respondents  fell  into  this  software  acquisition  model,  so  more  data  would  be  needed  to 
verify  if  this  holds  true  for  a  GOTS  software  acquisition  model. 
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Table  12.  Software  Acquisition  Model  (Question  3)  Compared  to  Specific 

Reviews  Included  (Question  6) 


- 

Q3:  COMMERCIAL  OFF  THE 

03:  GOVERNMENT  OFF  THE 

Q3:  NEW 

TOTAL 

SHELF  (A) 

SHELF  (B) 
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3 

1 

1 

5 
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1 

0 

1 

2 
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33.33% 
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-  Design  (3) 

2 

0 

1 

3 
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Bulls  Review 

66.67% 

2 

33.33% 

1 

0.00% 

a 

8.82% 

3 

-  Readress 

w 

Critical 

54.17% 

8.33% 

37.50% 

70.59% 

Design 

Review  (5) 

13 

2 

9 

24 

Re*»ng 

50.00% 

33.33% 

16.67% 

17.65% 
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Review  (6) 

3 

2 

1 

6 

FuncUonal 

50.00% 

20.00% 

30.00% 

29.41% 
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Audi  (7) 

c 

2 

3 

10 

irtagrascn 

44.44% 

0.00% 

55.56% 

26.47% 

ReadnesE 
Review  (8) 

4 

0 

5 

9 

Operasona 

50.00% 

25.00% 

25.00% 

35.29% 

Teel 

Rea dress 
Review  (9) 

6 

3 

3 

12 

Pltyslsal 

54.55% 

18.18% 

27.27% 

32.35% 

Corltguraosn 
Audi  (10) 

6 

2 

3 

11 

Praeidnary 

52.17% 

8.70% 

39.13% 

67.65% 

Desgr 

Review  (11) 

12 

2 

9 

23 

PToouseon 

54.55% 

0.00% 
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Readress 
Review  (12) 

6 

a 

5 

11 

Reease 

25.00% 

0.00% 

75.00% 

11.76% 

Bacwog 

Review  (13) 

1 

a 

3 

4 

SETR-Ulte 

40.00% 

40.00% 

20.00% 

14.71% 

Ergneertng 
Review  (14) 

2 

2 

1 

5 

Software 

28.57% 

14.29% 

57.14% 

20.59% 

speancawm 
Review  (15) 

2 

1 

4 

7 

System 

41.18% 

17.65% 

41.18% 

50.00% 

Funcflcnal 
Review  (16) 

7 

3 

7 

17 

System 

60.00% 

8.00% 

32.00% 

73.53% 

Requirements 
Review  (17) 

15 

2 

8 

2S 

System 

4286% 

7.14% 

50.00% 

41.18% 

VertltealWr 
Review  (IS) 

6 

1 

7 

14 

Tesl 

55  56% 

11.11% 
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79.41% 

Readress 
Review  (19) 

15 

3 

9 

27 

Oteer  (please 

70.00% 

10.00% 

20.00% 

29.41% 

specify)  (20) 

7 

1 

2 

10 

Response 

Responses 

Responses 

Tata 

20 

4 

10 

34 
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5. 


Subsidiary  Question  4 


The  fourth  subsidiary  question  is  “what  can  engineers  and  program  managers  of 
future  naval  acquisitions  learn  from  other  programs  that  have  attempted  to  tailor?”  The 
final  open  ended  survey  question  as  well  as  the  “other”  free  form  response  areas  in 
various  questions  provided  the  most  valuable,  direct  insight  to  address  this  question.  One 
survey  respondent  of  the  open-ended  question  included  the  following: 

While  SETR  is  a  great  tool  to  assess  where  a  program  is  in  the  time  of  the 
event.  We  are  normally  limited  in  time  and  resources  on  how  deep  to 
conduct  the  review.  We  are  also  limited  to  the  information  that  the 
programs  provide. 


I  had  an  instance  where  the  performers  provided  all  the  necessary 
information,  everything  appeared  on  schedule  for  delivery  and  the  SETR 
went  by  smoothly.  I  later  found  out  that  the  performers  were  11  months 
behind  schedule,  but  this  was  not  apparent  with  the  provided 
documentation. 


While  SETR  is  a  good  tool,  it  is  often  cumbersome  to  the  programs  to 

support  this  effort  while  still  performing  their  daily  tasks. 

Time  and  resource  constraints  are  key  reasons  that  tailoring  is  typically 
considered.  However,  as  noted  in  the  respondent’s  comment,  the  review  and  associated 
artifacts  being  evaluated  should  be  carefully  selected  to  hit  the  critical  areas  without 
adding  unnecessary  overhead  on  the  program,  avoid  causing  the  SETR  event  to  become  a 
“check-the-box”  event,  and/or  miss  risks  such  as  being  eleven  months  behind  schedule. 
Based  on  the  survey  results,  key  concepts  that  are  most  impactful  to  a  tailored  SETR 
implementation’s  success  are  as  follows: 

•  engage  leadership 

•  focus  on  preparation 

•  maintain  early  and  often  communication 

•  educate  stakeholders 

Leadership  engagement  came  up  repeatedly  through-out  the  survey  responses.  As 


Secretary  Kendall  states  in  his  description  of  the  optimal  program  structure,  the 
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individual’s  thinking  is  a  critical  piece,  which  requires  an  engaged  leader  (Kendall  2012, 
23).  The  program’s  SETR  leadership  should  seek  out  other  program  lessons  learned 
including  potential  review  templates  when  determining  how  they  are  going  to  structure 
the  implementation.  Similarly,  once  the  program  has  initiated  SETR  events,  the  program 
SETR  leadership  should  willingly  look  for  opportunities  to  share  with  other  programs  to 
ensure  best  practices  continue  to  evolve.  One  respondent  highlighted  a  leadership 
challenge  with  respect  to  engaging  and  understanding  the  variables  to  address  in  order  to 
avoid  events  becoming  “a  ‘check-the-box’  sort  of  event  if  the  program  is  required  to 
complete  every  single  SETR  review,  or  even  to  address  every  single  ‘bullet  point’  for  a 
given  review... SETR  reviews  are  absolutely  necessary,  but  forcing  programs  to  complete 
reviews  that  are  extraneous  given  their  specific  situation  leads  to  increased  costs, 
schedule,  etc.”  As  noted  in  the  literature  review,  the  PM  is  the  ultimate  recipient  of  the 
SETR  report  that  captures  the  confidence  level  of  the  baseline  (DON  SPAWAR  2009, 
1012).  The  manner  in  which  the  PM  addresses  the  report  sends  a  direct  message  to  the 
rest  of  the  program.  By  choosing  to  take  an  active  role  in  ensuring  action  items  or  other 
issues  are  addressed,  the  PM  lets  the  rest  of  the  program  office  know  that  SETR 
implementation  is  not  a  “check-the-box”  sort  of  event. 

SETR  events  require  focused  preparation,  which  is  probably  best  described  in  the 
following  SYSCOM  documents  that  provide  detailed  preparation  information  (e.g., 
scheduling,  review  elements)  for  each  review: 

•  NAVSEA  05H  Technical  Review  Manual ,  Version  2.0,  18  December 
2009:  Appendix  includes  detailed  descriptions  of  entrance  and  exit  criteria 
for  each  SETR  event,  different  documentation,  architectural  views,  and 
scheduling  details. 

•  NAVAIR  Adapting  Acquisition  to  Agile  Software  Development:  A  How- 
To  Guide ,  Version  2.0,  19  March  2014:  Provides  execution  level  details  on 
the  RBR. 

•  MCSC  SETR  Handbook  (SIAT-HDBK-001):  Detailed  appendix  for  each 
SETR  event  that  includes  an  event  overview,  entrance  criteria,  review 
elements,  agenda,  exit  criteria,  and  evaluation  criteria. 

As  noted  by  a  survey  respondent,  the  SETR  events  “typically  take  a  block  of 
preparatory  time  and  resolution  time  to  address  RFAs;”  however,  “the  benefit  is  that 
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multiple  SETR  events  ensure  documentation  is  being  updated  as  the  program  matures 
versus  the  documents  becoming  static  and  outdated.”  The  program  must  balance  focused 
preparation  with  ensuring  the  program  is  looking  at  what  events  will  provide  the  most 
ROI  for  the  program  such  as  the  PDR  or  CDR  prior  to  jumping  in  and  just  “checking  the 
box.” 

Early  and  often  communication  was  a  concept  that  developed  when  reviewing  the 
survey  responses  as  well  as  literature  review  to  enable  management  of  expectations  and 
reduce  risk  in  executing  a  tailored  SETR  event.  As  a  respondent  noted  “managing 
expectations  &  frequent  sync  sessions  are  critical  to  success.  Echelon  2  &  Echelon  3  have 
different  interpretations  of  what  is  expected  regarding  the  level  of  completeness  of  each 
SETR.”  Waiting  until  the  program  has  fully  vetted  internally  the  tailored  SETR 
implementation  without  reaching  out  to  external  stakeholders  increases  the  time  and 
chances  of  obstacles  being  put  in  the  program’s  path.  SETR  should  be  looked  at  as  a  way 
to  engage  users,  internal  and  external  stakeholders  to  deal  with  risks  before  they  become 
issues  in  an  open  transparent  manner,  which  requires  early  and  often  communication. 

Similarly,  educating  stakeholders  is  a  key  piece  to  ensuring  success  with  a 
tailored  SETR  implementation.  The  SETR  implementation  whether  tailored  or  not  must 
be  documented  in  the  program’s  SEP,  which  is  generally  signed  by  individuals  outside  of 
the  program  office.  Due  to  this,  if  the  program  determines  that  a  tailored  SETR 
implementation  is  best  then  the  program  must  educate  not  only  the  participants  of  the 
event,  but  also  external  stakeholder(s)  that  have  to  approve  the  SEP.  This  could  take 
significant  amount  of  time  as  one  respondent  noted,  “if  the  PM/LSE  uses  a  SETR  event 
that  senior  leadership/MDA  is  not  familiar  with,  then  it  may  result  in  additional 
exchanges  between  the  PM  and  senior  leadership/MDA  to  educate  both  sides  until  both 
sides  agree  on  how  it  will  be  addressed  and  executed.”  As  noted  in  the  literature  review 
section,  there  are  a  few  tailored  SETR  implementation  examples  documented  in  DOD, 
DON,  or  naval  SYSCOM  documents  to  lean  on  during  this  discussion,  which  one 
respondent  noted  “until  tailoring  is  addressed  directly  at  all  levels  from  policy  to  review 
guidelines,  attempting  to  tailor  will  lead  to  significant  friction,  fmstration,  and  delays”  as 
the  program  has  to  overcome  the  culture  of  what  people  find  acceptable.  Without 
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supportive  program  leadership,  overcoming  this  culture  could  prove  too  difficult  for  some 
SETR  leadership  teams  leading  to  the  “check-the-box”  mentality. 

C.  SUMMARY 

Leveraging  a  tailored  SETR  implementation  provides  the  necessary  structured 
engineering  framework  while  keeping  up  with  a  dynamic  software  development 
environment  to  meet  increasing  need  for  enhanced  capability  delivered  to  the  warfighter 
in  a  shorter  timeframe.  Over  80%  of  the  survey  respondents  from  across  various  DON 
organizations  indicated,  as  shown  in  Figure  15,  that  tailoring  was  occurring  within 
programs  to  address  program  specific  needs  such  as  aggressive  schedule  and  leveraging 
an  agile  software  development  models.  The  tailored  SETR  events  are  directly  impacting 
the  program’s  next  phase  through  technical  design  adjustments  and  other  key  program 
variables.  The  reviews  included  specifically  the  PDR  and  CDR,  as  well  as  entrance  and 
exit  criteria,  as  the  aspects  that  programs  find  provide  a  ROI.  This  is  the  case  for  both 
COTS  and  new  development  software  acquisition  models.  Factors  external  to  the 
organization  continue  to  be  the  primary  obstacle  no  matter  the  program  size  determining 
the  ability  to  successfully  tailor  and  implement  the  SETR  process  based  on  the  responses 
to  the  direct  survey  questions  and  open  ended  questions.  Future  naval  software 
acquisition  programs  should  engage  leadership,  focus  on  preparation,  maintain  early  and 
often  communication,  and  educate  stakeholders  to  improve  ROI  of  the  tailored  SETR 
implementation.  The  following  chapter  presents  the  conclusions  from  the  research  and 
suggestions  for  follow  on  research. 
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VI.  CONCLUSION 


A.  INTRODUCTION 

According  to  the  Naval  SYSCOM  Systems  Engineering  Policy  (2009),  the  SETR 
process  should  be  an  event-driven  process  that  consists  of  one  or  more  programmatically 
independent  reviews  with  defined  entrance  and  exit  criteria.  The  SETR  reviews  assess  the 
technical  health,  requirements  accuracy,  design  maturity,  testing  effectiveness,  and 
sustainment  support  over  the  program  life  cycle.  The  jointly  signed  SYSCOM  policy 
states  that  the  program  SETR  events,  along  with  the  event  entrance  and  exit  criteria,  are 
documented  in  the  SEP  which  is  signed  by  the  program  MDA  or  other  designated 
authority  based  on  the  program  ACAT  level.  The  SYSCOM  policy  notes  that  event 
closure  normally  occurs  only  after  the  exit  criteria  has  been  met,  but  the  SETR  TRB 
chair  must  concur  with  the  identified  action  items  along  with  any  plan  of  action  and 
milestones  and/or  mitigations.  According  to  the  policy,  the  SETR  output  ultimately 
informs  the  PM  if  the  program  is  technically  ready  to  move  on  to  the  next  phase  of  the 
acquisition  process. 

From  a  policy  perspective,  the  DOD  and  DON  directives  and  instructions  lay  the 
systems  engineering  foundation  for  the  naval  SYSCOM  guidance  regarding  SETR 
implementation.  In  addition,  the  Navy  has  emphasized  the  need  to  deliver  capability 
versus  systems,  and  acquisition  is  impacted  by  this  capability  vision  requiring  innovative 
application  of  SETR  for  the  SoS  or  platform  level  efforts,  which  traditional  SETR  is  not 
well  structured  to  support.  Leveraging  a  tailored  SETR  implementation  provides  the 
necessary  structured  engineering  framework  while  keeping  up  with  a  dynamic  software 
development  environment  to  meet  increasing  need  for  enhanced  capability  delivered  to 
the  warfighter  in  a  shorter  timeframe.  More  than  80%  of  the  survey  respondents  from 
across  various  DON  organizations  indicated,  as  shown  in  Figure  15,  that  tailoring  was 
occurring  within  programs  to  address  program  specific  needs  such  as  aggressive  schedule 
and  leveraging  an  agile  software  development  model.  The  tailored  SETR  events  are 
directly  impacting  the  program’s  next  phase  through  technical  design  adjustments  and 
other  key  program  variables.  The  reviews  included  specifically  the  PDR  and  CDR,  as 
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well  as  entrance  and  exit  criteria,  as  the  aspects  that  programs  find  provide  a  ROI  when 
tailoring  the  SETR  implementation.  This  is  the  case  for  both  COTS  and  new  development 
software  acquisition  models.  Factors  external  to  the  organization  continue  to  be  the 
primary  obstacle  no  matter  the  program  size  determining  the  ability  to  successfully  tailor 
and  implement  the  SETR  process. 

B.  RECOMMENDATIONS 

Future  naval  software  acquisition  programs  should  engage  leadership,  focus  on 
preparation,  maintain  early  and  often  communication,  and  educate  stakeholders  to 
improve  ROI  of  the  tailored  SETR  implementation.  As  Secretary  Kendall  states  in  his 
description  of  the  optimal  program  structure,  the  individual’s  thinking  is  a  critical  piece, 
which  requires  engaged  leaders  (Kendall  2012,  23).  Seeking  out  other  program  lessons 
learned  prior  to  determining  how  to  tailor  the  program’s  SETR  implementation  will 
reduce  the  likelihood  of  tailored  implementations  that  provide  no  ROI  on  the  program’s 
SETR  implementation. 

Similarly,  once  the  program  has  initiated  SETR  events  the  program  SETR 
leadership  should  willingly  look  for  opportunities  to  share  with  other  programs  to  ensure 
best  practices  continue  to  evolve.  The  program  must  balance  focused  preparation  with 
ensuring  the  program  is  including  reviews  in  the  tailored  implementation  that  will 
provide  the  most  ROI  for  the  program  (such  as  the  CDR)  prior  to  jumping  in  and  just 
“checking  the  box.” 

Educating  stakeholders  is  a  key  piece  to  ensuring  success  with  a  tailored  SETR 
implementation.  Additionally,  waiting  until  the  program  has  fully  vetted  internally  the 
tailored  SETR  implementation  without  reaching  out  to  communicate  with  external 
stakeholders  increases  the  time  and  chances  of  obstacles  being  put  in  the  program’s  path. 
SETR  should  be  looked  at  as  a  way  to  engage  users,  internal  and  external  stakeholders  to 
deal  with  risks  before  they  become  issues  in  an  open  transparent  manner,  which  requires 
early  and  often  communication.  The  SETR  implementation,  whether  tailored  or  not,  must 
be  documented  in  the  program’s  SEP,  which  is  generally  signed  by  individuals  outside  of 
the  program  office.  Due  to  this,  if  the  program  determines  that  a  tailored  SETR 
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implementation  is  best,  then  the  program  must  educate  not  only  the  participants  of  the 
event,  but  also  external  stakeholders  that  have  to  approve  the  SEP.  This  could  take 
significant  amount  of  time.  As  noted  in  the  literature  review  section,  there  are  a  few 
tailored  SETR  implementation  examples  documented  in  DOD,  DON,  or  naval  SYSCOM 
documents  to  lean  on  during  this  discussion.  Without  supportive  program  leadership, 
overcoming  this  culture  could  prove  too  difficult  for  some  SETR  leadership  teams 
leading  to  the  “check-the-box”  mentality. 

From  a  policy  prospective,  tailoring  needs  to  be  addressed  directly  at  all  levels  to 
avoid  significant  friction,  frustration,  and  delays.  Ideally,  the  SYSCOM  guidance  or 
supplemental  documentation  should  provide  examples  of  successfully  tailoring  and 
lessons  learned  that  future  naval  programs  can  use  as  a  starting  point  to  avoid  having  to 
learn  the  lessons  the  hard  way.  Additionally,  the  SYSCOM  level  documentation  should 
address  minimal  level  metrics  to  objectively  measure  SETR  implementation  ROI  to 
ensure  the  events  are  not  “check-the-box”  events,  which  only  serve  to  drain  already 
constrained  resources  within  the  program. 

C.  FUTURE  RESEARCH 

As  the  DON  continues  to  focus  priorities  around  delivering  capabilities  versus 
individuals  systems,  additional  research  to  improve  the  SETR  implementation  across  the 
department  is  critical,  especially  to  stay  current  with  industry  software  best  practices, 
which  is  leading  to  more  rapid  software  deployments.  Then-Secretary  Gates  framed  the 
challenge  in  a  September  2008  speech.  “Our  conventional  modernization  programs  see  a 
99%  solution  in  years.  Stability  and  counterinsurgency — the  wars  we  are  in — require  a 
75%  solution  in  months.  The  challenge  is  whether  in  our  bureaucracy  and  in  our  minds 
these  two  different  paradigms  can  be  made  to  coexist”  (SEI  2011,  xi).  Warfighters  always 
benefit  from  finding  ways  to  reduce  the  bureaucracy  and  deliver  enhanced  capabilities  in 
a  shorter  timeframe  addressing  the  dynamic  threats  they  face  daily.  The  alternative  to 
allowing  bureaucracy  to  triumph,  from  a  DON  perspective,  has  potential  consequences 
such  as  loss  of  life  that  should  motivate  every  individual  involved  in  the  SETR 
implementation  to  achieve  the  greatest  ROI  in  the  shortest  amount  of  time. 
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More  than  80%  of  survey  respondents  stated  that  the  program  did  not  include 
metrics  to  assess  the  ROI  of  the  SETR  reviews.  From  a  literature  review  perspective, 
SPAWAR  Systems  Engineering  Technical  Review  Organization  Standard  Process 
Handbook  was  the  only  document  that  addressed  metrics  beyond  a  brief  mention. 
Research  identifying  metrics  to  assess  ROI  would  not  only  help  future  programs  from  a 
lesson  learned  perspective,  but  also  could  help  programs  executing  the  SETR  event 
understand  how  to  adjust  future  reviews  gaining  more  value  on  what  can  be  delivered  to 
the  warfighter.  Another  related  research  area  is  the  average  SETR  investment  from  the 
program  as  it  relates  to  the  average  ROI  (e.g.,  cost  savings,  schedule  savings).  This 
research  would  benefit  holistically  from  additional  statistical  analysis  to  better  tie  the 
return  on  investment  to  the  type  of  tailoring  that  was  implemented. 

Beyond  metrics,  an  area  that  developed  during  this  research  due  to  DODI  5000.75 
being  published  was  the  difference  in  a  SETR  event  for  a  business  system  versus  a 
national  security  system.  DODI  5000.75  supersedes  DODI  5000.02  for  all  business 
system  acquisition,  which  directs  alignment  “to  commercial  best  practices  and  will 
minimize  the  need  for  customization  of  commercial  products  to  the  maximum  extent 
possible”  (DOD  2017,  4).  This  should  lead  to  more  accepted  use  of  SETR  tailoring, 
which  will  provide  opportunities  to  further  this  research. 

D.  SUMMARY 

This  research  provides  tailored  SETR  implementation  recommendations  based  on 
lessons  learned  on  how  to  increase  positive  ROI  for  a  naval  software  acquisitions 
program.  The  recommendations  will  assist  program  leadership  in  making  better  decisions 
on  where  to  allocate  software  engineering  resources  within  the  schedule  and  funding 
constraints.  While  the  thesis  is  focused  on  the  DON,  the  recommendations  are  applicable 
to  SETR  implementations  for  software  acquisitions  programs  across  the  broader  DOD.  In 
addition,  three  future  research  topics  are  provided  based  on  areas  that  came  up  in  this 
effort  that  could  be  expanded  upon.  Tailored  SETR  implementation  has  not  been  heavily 
researched  specifically  as  it  relates  to  software  acquisition  programs,  so  there  is  a 
significant  opportunity  to  positively  impact  the  product  the  warfighter  receives. 
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APPENDIX  A.  LEADING  INDICATORS 


The  following  tables  show  leading  indicators  in  relation  to  IEEE  Standard  15288. 


Table  4  - 

LEADING  INDICATORS  APPLICATION 

PER 

ISO/IEC  15288 

Requirements  Trends 

System  Definition  Change 

Backlog  Trend 

Interface  Trends 

Requirements  Validation 

Trends 

Requirements 

Verification  Trends 

Work  Product  Approval 

Trends 

Review  Action  Closure 

Trends 

Risk  Exposure  Trends 

Risk  Treatment  Trends 

Technology  Maturity 

Trends 

Technical  Measurement 

Trends 

Systems  Engineering 

Staffing  &  Skills  Trends 

Process  Compliance 

Trends 

Test  Completeness 

Trends 

Facility  and  Equipment 

Availability  Trends 

Defect/ Error  Trends 

System  Affordability 

Trends 

Architecture  Trends 

Schedule  and  Cost 

Pressure 

6.3  Project  Processes 

6.3.1  Project  Planning  Process 

6.3.1. 3.a  Define  the  project 

6.3.1. 3.b  Plan  the  project  resources 

X 

X 

X 

6.3.1. 3.c  Plan  the  project  technical  and  quality 
management 

X 

X 

X 

6.3.1. 3.d  Activate  the  project 

6.3.2  Project  Assessment  and  Control  Process 

6.3.2.3.a  Assess  the  project 

X 

X 

X 

X 

X 

X 

X 

6.3.2.3.b  Control  the  project 

X 

X 

X 

X 

X 

X 

X 

6.3.2.3.C  Close  the  project 

6.3.3  Decision  Management  Process 

6.3.3.3.a  Plan  and  define  decisions 

X 

X 

6.3.3.3.P  Analyze  the  decision  information 

X 

X 

6.3.3.3.C  Track  the  decision 

X 

X 

6.3.4  Risk  Management  Process 

6.3.4.3.a  Plan  Risk  Management 

6.3.4.3.b  Manage  Risk  Profile 

6.3.4.3.C  Analyze  Risks 

X 

6.3.4.3.d  Treat  Risks 

X 

X 

6.3.4.3.e  Monitor  Risks 

X 

X 

6.3.4.3.f  Evaluate  Risk  Management  Process 

X 

X 

6.3.5  Configuration  Management  Process 

6.3.5.3.a  Plan  configuration  management 

6.3.5.3.b  Perform  configuration  management 

X 
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Table  4  -  LEADING  INDICATORS  APPLICATION 

PER 

ISO/IEC  15288 

Reguirements  Trends 

System  Definition  Change 

Backlog  Trend 

Interface  Trends 

Reguirements  Validation 

Trends 

Reguirements 

Verification  Trends 

Work  Product  Approval 

Trends 

Review  Action  Closure 

Trends 

Risk  Exposure  Trends 

Risk  T  reatment  T  rends 

Technology  Maturity 

Trends 

Technical  Measurement 

Trends 

Systems  Engineering 

Staffing  &  Skills  Trends 

Process  Compliance 

Trends 

Test  Completeness 

Trends 

Facility  and  Eguipment 

Availability  Trends 

Defect/ Error  Trends 

System  Affordability 

Trends 

Architecture  Trends 

Schedule  and  Cost 

Pressure 

6.3.6  Information  Management  Process 

6.3.6.3.a  Plan  Information  management 

6.3.6.3.b  Perform  information  management 

X 

6.4  Technical  Processes 

6.4.1  Stakeholder  Reguirements  Definition 
Process 

6.4.1. 3.a  Elicit  Stakeholder  Reguirements 

X 

6.4.1. 3.b  Define  Stakeholder  Reguirements 

| 

X 

6.4.1.3.C  Analyze  and  Maintain  Stakeholder 
Reguirements 

X 

X 

X 

X 

X 

6.4.2  Reguirements  Analysis  Process 

6.4.2.3.a  Define  System  Reguirements 

X 

X 

6.4.2.3.b  Analyze  and  Maintain  System 
Reguirements 

X 

X 

X 

X 

X 

X 

6.4.3  Architectural  Design  Process 

6.4.3.3.a  Define  Architecture 

X 

X 

X 

X 

6.4.3.3.b  Analyze  and  Evaluate  Architecture 

X 

X 

X 

X 

X 

6.4.3.3.C  Document  and  Maintain  Architecture 

X 

X 

X 

6.4.4  Implementation  Process 

6.4.4.3.a  Plan  implementation 

6.4.4.3.b  Perform  implementation 

X 

6.4.5  Integration  Process 

6.4.5.3.a  Plan  integration 

X 

6.4.5.3.b  Perform  integration 

X 

X 

6.4.6  Verification  Process 

6.4.6.3.a  Plan  verification 

X 

X 

Table  4  -  LEADING  INDICATORS  APPLICATION  PER  ISO/IEC  15288 

Reguirements  Trends 

System  Definition  Change 

Backlog  Trend 

Interface  Trends 

Reguirements  Validation 
Trends 

Reguirements 
Verification  Trends 

Work  Product  Approval 
Trends 

Review  Action  Closure 
Trends 

Risk  Exposure  Trends 

Risk  T  reatment  T  rends 

Technology  Maturity 
Trends 

Technical  Measurement 
Trends 

Systems  Engineering 
Staffing  &  Skills  Trends 

Process  Compliance 
Trends 

Test  Completeness 
Trends 

Facility  and  Eguipment 

Availability  Trends 

Defect/ Error  Trends 

System  Affordability 

Trends 

Architecture  Trends 

Schedule  and  Cost 

Pressure 

6.4.6.3.b  Perform  verification 

X 

X 

6.4.8  Validation  Process 

6.4.8.3.a  Plan  validation 

X 

X 

6.4.8.3.b  Perform  validation 

X 

X 

6.4.10  Maintenance  Process 

6.4.10.1a  Plan  maintenance 

X 

6.4.10.1b  Perform  maintenance 

X 
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APPENDIX  B.  COMPLETE  SURVEY  DATA 


Question  1:  Which  organization  are  you  most  closely  associated  with? 

Answer  Options 

Response 

Percent 

Response 

Count 

Department  of  Defense 

4.2% 

2 

Department  of  Navy 

27.1% 

13 

Marine  Corps  Systems  Command 

6.3% 

3 

Naval  Air  Systems  Command 

2.1% 

1 

Naval  Facilities  Engineering  Command 

0.0% 

0 

Naval  Sea  Systems  Command 

2.1% 

1 

Naval  Supply  Systems  Command 

2.1% 

1 

Space  and  Naval  Warfare  Systems  Command 

54.2% 

26 

Other 

2.1% 

1 

answered  question 

48 

skipped  question 

0 

Question  2:  What  is  the  size  of  the  program  that  used  the  SETR  process? 

Answer  Options 

Response 

Percent 

Response 

Count 

ACATI 

35.4% 

17 

ACAT  II 

2.1% 

1 

ACAT  III 

22.9% 

11 

Other  (please  specify) 

39.6% 

19 

answered  question 

48 

skipped  question 

0 

Question  2:  Other  (please  specify) 

1 

MCSC/PEO  LS  acquisition  programs  range  from  ACAT  I  down  to  AAPs 

2 

Not  ACAT.  Program  is  less  than  $3  million. 

3 

Non-POR 

4 

Non-Program  of  record 

5 

Project 

6 

Most  Legacy  systems  don’t  use  SETR 

7 

One  POR  consisting  of  multiple  abbreviated  acquisition  programs  (AAP);  ACAT 

ITT  equivalent  overall 

8 

AAP  and  an  ACAT  III  program 

9 

ACAT  4  and  Non-ACAT  programs 
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10 

Abbreviated  Acquisition  Program  (most  recent) 

11 

Smaller  programs  that  support  ACAT  I- III  programs 

12 

abbreviated  acquisition  systems 

13 

AAP 

14 

AAP 

15 

ACAT  IV  (T) 

16 

Not  a  program  of  record;  an  R&D  program  pre-milestone  B  equivalent  to  ACAT  I 

17 

R&D  Demonstration  Program 

18 

Demonstration  Program  vice  Acquisition  program. 

19 

All  ACAT  level  programs  within  PEO  C4I  go  through  some  SETR  events. 

Question  3:  What  type  of  software  acquisition  model  did  this  program  leverage? 

Answer  Options 

Response 

Percent 

Response 

Count 

Commercial  off-the-shelf 

43.8% 

21 

Government  off-the-shelf 

8.3% 

4 

New  Development 

20.8% 

10 

Other  (please  specify) 

27.1% 

13 

answered  question 

48 

skipped  question 

0 

Question  3:  Other  (please  specify) 

1 

Maximum  use  of  COTS,  with  some  defined  use  of  GOTS. 

2 

Depends  on  the  acquisition  program 

3 

Non  Program  of  Record  (Non-POR) 

4 

Rapid  IT  Acquisition — COTS 

5 

In  house  Development  years  ago.  Being  maintained  but  not  suitable  to  SETR, 
being  phased  out  if  they  can  ever  find  a  COT  fit  (but  is  not  advisable,  at  all,  in  my 
book). 

6 

A  combination  of  COTS,  GOTS,  and  new  development  followed  by  an  Integration 
activity. 

7 

both  COTS/GOTS 

8 

Mix  of  COTS  (Office-type  products),  GOTS  (For  architecture),  and  New  Plug-in 
Development 

9 

Maintenance  of  existing  SPAWAR-developed  software  plus  some  new 
development. 

10 

Some  new  development  with  large  amount  of  integration  with  COTS  and  GOTS 

11 

All  of  the  above. 

12 

We  deal  with  all  of  these  activities. 
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I  don’t  understand  this  question. 


13 


On  one  effort  we  used  the  IT  Box  process,  on  others  we  tailored  to  meet  our  needs 
for  the  acquisition  process. 


Question  4:  Did  the  program  tailor  the  system  engineering  technical  review 
process? 

Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

81.3% 

39 

No 

18.8% 

9 

answered  question 

48 

skipped  question 

0 

Question  5:  If  applicable,  which  part  did  you  tailor? 

Answer  Options 

Response 

Percent 

Response 

Count 

Review  Order 

13.6% 

6 

Reviews  Included 

70.5% 

31 

Entrance/Exit  Criteria 

65.9% 

29 

Not  Applicable 

15.9% 

7 

Other  (please  specify) 

18.2% 

8 

answered  question 

44 

skipped  question 

4 

Question  5:  Other  (please  specify) 

1 

All  of  the  above.  Each  acquisition  program  is  unique  and  tailors  it  program  based 
on  many  factors. 

2 

Those  entrance  and  exit  criteria  that  does  not  apply  to  non-ACAT  or  Non-POR. 

3 

Required  Acquisition  Documentation 

4 

Also  combined  multiple  events  into  one  (i.e.  SRR/SFR,  instead  of  two  separate 
reviews/events);  held  “light”  versions  of  various  events  (e.g.,  PDR,  CDR)  after 
major  requirements  and  design  overhauls 

AAP  tailored  the  entrance/exit  criteria  for  all  SETR  events 

5 

ACAT  III  has  tailored  all  SETR  events  to  the  point  where  the  effectiveness  of 
events  have  been  compromised. 

6 

literally  everything.  The  existing  SPAWAR  SETR  process  is  wholly  overbearing 
and  useless  to  line  engineers.  Had  to  rescope  our  review  of  contractor  deliverables 
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to  a  very  simple  Requirements  Review  +  Initial  Design  Review  (aka  PDR)  +  Final 
Design  Review  (aka  CDR)  +  Testing  Review  (TRR  +  Test  Results  all  rolled  into 
one). 


7 

Scope  of  documentation  and  reviews.  The  existing  software  was  developed  before 
the  SETR  process  applied;  thus,  not  all  required  documentation  now  required  exists. 
Because  the  program  is  in  sustainment,  funding  to  retroactively  develop  the 
documentation  is  not  available. 

8 

Applicable  documentation.  For  my  demonstration  program  the  systems  engineering 
process  is  focused  on  requirements  definition  with  less  focus  on  life  cycle 
engineering  as  that  is  not  prudent  unless  the  program  transitions  to  a  full  scale 

ACAT  program. 

Question  6:  What  reviews  did  the  program  include  in  the  SETR  process? 

Answer  Options 

Response 

Percent 

Response 

Count 

Build  Technical  Review 

17.0% 

8 

Capability  Build  Review — Initial 

4.3% 

2 

Capability  Build  Review — Design 

6.4% 

3 

Capability  Build  Review — Readiness 

6.4% 

3 

Critical  Design  Review 

68.1% 

32 

Fielding  Technical  Review 

17.0% 

8 

Functional  Configuration  Audit 

27.7% 

13 

Integration  Readiness  Review 

21.3% 

10 

Operational  Test  Readiness  Review 

36.2% 

17 

Physical  Configuration  Audit 

29.8% 

14 

Preliminary  Design  Review 

68.1% 

32 

Production  Readiness  Review 

29.8% 

14 

Release  Backlog  Review 

10.6% 

5 

SETR-Fite  Engineering  Review 

14.9% 

7 

Software  Specification  Review 

19.1% 

9 

System  Functional  Review 

46.8% 

22 

System  Requirements  Review 

70.2% 

33 

System  Verification  Review 

36.2% 

17 

Test  Readiness  Review 

76.6% 

36 

Other  (please  specify) 

36.2% 

17 

answered  question 

47 

skipped  question 

1 

Question  6:  Other  (please  specify) 

1 

Flight  Readiness  Review 
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2 

What  SETR  reference  is  being  used  for  the  list  provided?  Some  programs  create 
their  own  unique  technical  reviews.  There  are  other  SETRs  that  are  not  in  the  list 
provided. 

3 

System  Requirements  Review  and  System  Functional  Review  were  combined. 

4 

ATO,  ATP,  CRD,  FOT&T,  IBR,  ILA,  PD&E,  RFI,  RFP,  UAT 

5 

Navy  HR  Pers/Pay  systems  have  so  many  unique  requirements  that  ongoing 
maintenance  will  be  cheaper  than  conversion  and  the  cost  of  new  licenses, 
customization,  upgrades,  new  licenses,  re-customizations  to  the  upgrades  and  that 
repetitive  cycle  over  and  over  through  time. 

6 

TRR  was  followed  by  independent  component  and  integration  testing. 

7 

Our  initial  Engineering  Master  Plan  called  out  combining  2  SETR  events  into 
each  review,  with  a  total  of  6  reviews. 

AAP  tailored  entrance/exit  criteria  for  SRR,  PDR,  CDR,  PRR,  FCA,  SVR,  PCA, 
TRR. 

8 

ACAT  III  focus  is  on  speed  to  capability  and  is  using  SETR-Lite  Engineering 
Reviews  to  accelerate  process. 

9 

The  SRR  and  PDR  was  combined  into  a  single  event 

10 

Combined  SRR/SFR 

11 

Non-Development  Item  Integration  Review  (NIR)  in  lieu  of  PDR/CDR  for 
integration  of  COTS  products. 

12 

USMC-specific  Non-Developmental  Item  (NDI)  Integration  Review  (NIR)  which 
is  comparable  to  a  CDR  but  for  NDI/COTS  items.  See  MCSC  Technical  Review 
Handbook. 

13 

Other  review  were  conducted,  but  not  while  I  was  on  the  project 

14 

NIR  also  included. 

15 

OTRR 

16 

Many  of  the  elements  of  the  non-implemented  reviews  are  captured  via  quarterly 
program  review  events.  Regular  course  check  on  technical  progress  vice  milestone 
reviews.  Also,  contractors  handle  a  lot  of  the  systems  engineering  milestones  at 
their  level.  Some  reviews  are  held  with  government  participation  and  insight,  but 
not  a  government  approval  panel. 

17 

We  also  in  what  I  believe  is  called  an  In-process  review  (I  dont  recall  off  hand  but 
it  is  a  mini  review  of  the  effort  at  a  given  time — between  other  SETR  reviews). 

Not  sure  if  that  qualifies  but  it  acts  like  a  PMR  but  focused  on  engineering  efforts. 

Question  7:  Which  review  events  did  the  program  find  most  valuable  to  assessing 
the  technical  maturity  of  the  program? 

Answer  Options 

Response 

Percent 

Response 

Count 

Build  Technical  Review 

4.2% 

2 

Capability  Build  Review — Initial 

4.2% 

2 

Capability  Build  Review — Design 

8.3% 

4 

Capability  Build  Review — Readiness 

6.3% 

3 
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Critical  Design  Review 

56.3% 

27 

Fielding  Technical  Review 

4.2% 

2 

Functional  Configuration  Audit 

4.2% 

2 

Integration  Readiness  Review 

12.5% 

6 

Operational  Test  Readiness  Review 

12.5% 

6 

Physical  Configuration  Audit 

8.3% 

4 

Preliminary  Design  Review 

39.6% 

19 

Production  Readiness  Review 

12.5% 

6 

Release  Backlog  Review 

4.2% 

2 

SETR-Lite  Engineering  Review 

6.3% 

3 

Software  Specification  Review 

10.4% 

5 

System  Functional  Review 

12.5% 

6 

System  Requirements  Review 

29.2% 

14 

System  Verification  Review 

16.7% 

8 

Test  Readiness  Review 

39.6% 

19 

Other  (please  specify) 

12.5% 

6 

answered  question 

48 

skipped  question 

0 

Question  7:  Other  (please  specify) 

1 

Technical  maturity  is  continuously  assessed  along  the  acquisition  life  cycle. 

2 

NSIPS  was  estimated  to  require  no  more  than  15  to  20  to  install  initially,  but 
required  90%  change  to  implement.  It  has  been  required  to  upgrade  with  new 
licenses,  re-customization,  and  requires  a  30  day  period  to  complete  end  to  end 
testing  of  changes.  Legacy  can  change  in  1  to  2  days  including  testing  all 
requirements.  Navy  Legacy  HR  data  is  on  mainframe;  not  ever  hacked,  New 
systems  want  to  use  “cloud”  technology,  which  can  be  hacked.  Navy  members 
should  never  be  in  a  setting  that  is  known  to  be  easy  to  hack.  Veterans/Service 
Members  should  know  then-  HR  data  is  safe,  not  subject  to  any  hacking;  veterans 
and  current  members  deserve  to  have  their  data  locked  up  and  protected  to  the  best 
of  our  ability  to  do  that  !  !  ! 

3 

Our  Program  is  still  Pre-Milestone  B  (ATP)  and  in  RFP  release  stage.  We  have  only 
been  through  CDR  and  SRR.  Our  next  scheduled  SETR  is  not  until  Q2  FY19. 

4 

NIR 

5 

Did  not  find  any  valuable.  Only  time  consuming. 

6 

Quarterly  Program  Reviews. 

Question  8:  What  area  of  the  program  was  adjusted  based  on  the  outcome  of  the 
SETR? 

Answer  Options 

Response 

Percent 

Response 

Count 
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Cost 

29.2% 

14 

Schedule 

50.0% 

24 

Technical  Design 

52.1% 

25 

No  change 

22.9% 

11 

Other  (please  specify) 

20.8% 

10 

answered  question 

48 

skipped  question 

0 

Question  8:  Other  (please  specify) 

1 

Various  areas  may  be  adjusted  based  on  the  outcome  of  the  SETR.  However,  the 
outcomes  are  based  on  PM  decisions  since  SETR  close-out  reports  are  from  the 
program-independent  SETR  chair  and  to  the  PM.  The  SETR  close-out  report  will 
include  recommendations  for  the  PM  to  consider. 

2 

Depending  on  what  was  going  to  be  available  or  ready  for  delivery,  what  will  be 
included  in  the  release  package. 

3 

Engineers  want  computers  and  computer  science  codified  to  engineering  standards, 
and  while  some  of  those  concepts  apply,  some  don’t.  Computer  science,  to  have  as 
a  tool  for  improving  office  systems  and  business  systems  is  not  the  same  as 
(similar,  but  different)  computers  for  robotic  type  of  situations,  or  warfare  types  of 
situations.  No  amount  of  training  (in  engineering  steps)  can  replace  a  well 
qualified  IS/IT/ADP  top-of-the  line  specialist.  Engineers  will  try  to  prove  they  can 
over  and  over,  but  that  relies  on  the  concept  of  saying  one  size  fits  all  (which  is 
true  in  some  cases)  which  does  not  expand  on  the  qualified  IT  Specialist’s  ability 
to  adopt/adept/fits  right,  the  needs  of  a  business  to  a  particular  situation,  i.e., 

Military  HR  PERS/PAY  are  unique  but  must  be  a  fit  to  serve  each  service  to  fulfill 
the  needs  of  all  applicable  laws  Congress  can  pass. 

4 

Contract  modifications  occurred  as  result  of  the  SETR  Design  reviews  discovering 
that  there  was  lack  of  interoperability  with  fielded  capabilities.  Additionally,  the 
Design  Reviews  produced  reprioritization  of  the  interfaces. 

5 

System  Requirements  Specification 

For  AC  AT  III — no  change 

6 

for  AAP — Cost,  Schedule  and  technical  design 

7 

SETR  had  no  impact  on  what  the  PM  wanted  to  do.  He  did  what  he  wanted  to  do 
regardless  of  what  the  output  of  the  SETR  said. 

8 

Typically  design  was  refined — not  changed.  Significant  design  changes  occurred 
later  during  implementation  that  caused  substantial  delay.  The  NIR  was  treated  by 
leadership  as  a  check-in-the-box  event  rather  than  a  substantial  design  review — 
perhaps  the  design  issues  could  have  been  avoided  if  treated  otherwise. 

9 

Cost  more  /  schedule  ended  up  much  longer. 

10 

Reviews  were  so  tailored  in  terms  of  entry  /exit  criteria,  that  the  reviews  were 
driven  to  ensure  passing.  This  meant  all  risks  associated  were  deemed  “acceptable” 
and  basically  transformed  into  issues. 
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Question  9:  By  what  percentage  of  the  cost,  schedule,  and/or  technical  design  was 
the  program  adjusted,  if  applicable? 

Answer  Options 

Response 

Percent 

Response 

Count 

0-20% 

56.3% 

27 

21-40% 

8.3% 

4 

41-60% 

12.5% 

6 

61-80% 

0.0% 

0 

81-100% 

0.0% 

0 

Not  Applicable 

22.9% 

11 

answered  question 

48 

skipped  question 

0 

Question  10:  Did  any  of  the  following  options  limit  or  create  obstacles  if  the 
program  tailored  the  SETR  implementation? 

Answer  Options 

Response 

Percent 

Response 

Count 

Internal  Organization 

31.3% 

15 

External  Organization 

56.3% 

27 

Policy 

18.8% 

9 

No  limitation 

12.5% 

6 

Not  applicable 

12.5% 

6 

Other  (please  specify) 

16.7% 

8 

answered  question 

48 

skipped  question 

0 

Question  10:  Other  (please  specify) 

1 

It  depends.  For  instance,  if  the  PM/LSE  uses  a  SETR  event  that  senior  leadership/ 
MDA  is  not  familiar  with,  then  it  may  result  in  additional  exchanges  between  the 

PM  and  senior  leadership/MDA  to  educate  both  sides  until  both  sides  agree  on  how 
it  will  be  addressed  and  executed. 

2 

Every  Navy  ‘Heavy-weight’  that  controls  money  in  the  budget  thinks  they  can 
become  a  computer  guru  by  buying  the  latest  and  greatest  product  that  a  salesperson 
can  talk  them  into,  and  don’t  seem  to  realize  that  all  requirements  of  ANY  systems 
should  be  known  PRIOR  TO  any  purchase.  Money  to  spend  doesn’t  make  a 
manager  an  expert,  and  they  will  be  rotated  out  of  that  job  in  (usually)  three  years 
(or  less)  when  someone  else  will  come  in  and  make  decisions  all  over  again.  If  you 
don’t  know  how  a  present  system  runs  (to  fulfill  all  the  laws,  policies,  required 
today),  then  how  can  you  say  that  you  are  an  expert  qualified  to  replace  that  system. 

If  you  force  the  conversion  on  partial  knowledge  then  the  risk  of  failure  increases 
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and  all  members  of  that  service  suffer  and  Congress  is  unhappy.  Why?  Because 
they  controlled  the  budget  and  had  the  authority  but  did  not  have  sufficient 
knowledge,  or  training,  or  experience,  to  make  the  right  call  knowing  the  risks. 

Personnel —  The  organization  was  not  staffed  with  the  appropriate  number  of 
people. 

3 

Skillset  -The  work  force  was  not  trained  for  the  positions  they  held. 

4 

PMW  240,  as  a  portfolio  of  smaller  programs,  is  built  around  the  idea  that  the 

SETR  process  needs  to  be  tailored  for  each,  individual  program;  leadership  is  also 
aware  that  defense  business  systems,  especially  unmodified  COTS 
implementations,  are  unique  among  DOD  acquisitions,  which  necessitates 
modifications  to  the  SETR  process 

The  Program  Manager  did  not  and  currently  does  not  care  for  engineers.  They  “get 
in  his  way.” 

5 

Maybe  that’s  why  his  program  is  failing...? 

6 

Actual  SETR  events  were  determined  by  MCSC  leadership  external  to  the  Program 
Office. 

7 

There  is  very  few  tailoring  guidelines.  This  leads  to  significant  disagreement  about 
what  tailoring  is  acceptable.  This  problem  exists  at  all  levels.  Take  architecture  for 
example:  the  DoDAF  specifications  are  vague  on  tailoring,  the  Navy  Architecture 
Development  Guide  does  not  directly  address  tailoring,  and  the  SPAWAR 
architecture  review  guidelines  (used  by  contractors  to  conduct  reviews)  make  no 
provision  for  tailoring.  Until  tailoring  is  addressed  directly  at  all  levels  from  policy 
to  review  guidelines,  attempting  to  tailor  will  lead  to  significant  friction,  frustration, 
and  delays. 

8 

The  external  organizations  (user  and  customer)  were  not  involved  in  the  review. 
Obstacles  arose  in  the  approval  of  a  design  that  the  user  had  never  seen  and  making 
design  decisions  driving  cost  and  schedule  that  the  customer  did  not  have  input  to 
change. 

Question  11:  Did  the  program  include  any  metrics  to  assess  the  return  on 
investment  from  the  reviews? 

Answer  Options 

Response 

Percent 

Response 

Count 

No 

80.9% 

38 

Yes  (please  specify) 

19.1% 

9 

answered  question 

47 

skipped  question 

1 
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Question  11:  Yes  (please  specify) 

1 

Development,  Test,  Cost  metrics 

2 

This  should  be  done  on  the  operating  for  the  thousands  of  PCs  the  Navy,  of  MS 
licenses  versus  the  cost  of  Linux  type  (that  can  be  make  more  secure  than  MS 
windows).  We  could  save  thousands,  if  not  millions,  or  billions,  and  that  money 
could  go  for  personnel  “perks,”  ships  and  weapon  systems  if  only  they  looked  at  the 
long  range  rather  than  see  how  much  of  a  budget  they  could  control,  or  how  much 
money  (of  that  budget)  they  could  obligate  and  spend! 

3 

Earned  Value  Management  (EVM) 

Action  Items  tracking , 

Tailored  Entrance  and  Exit  Criteria 

Interface  prioritization  spreadsheet 

Testing  Defects  per  build 

Functional  and  Nonfunctional  Requirements  per  build 

4 

Initial  business  case  and  subsequent  validation. 

5 

The  SETR  events  were  viewed  as  program  milestones  necessary  for  moving 
forward  with  the  next  phase  of  the  project.  Each  SETR  event  resulted  in  RFAs  that 
held  the  SETR  event  open  until  the  RFAs  could  be  addressed  and  answered. 

6 

AoAs  were  run  at  critical  decisions  in  the  program.  COAs  were  used  to  provide  the 
ROI  and  overall  benefits  or  constraints.  Selected  CO  A  implemented  with  MDA  sign 
off. 

7 

We  developed  a  comment  review  process  that  tracks  comments  using  our  CM  tool 
(IBM  JAZZ  RTC).  This  allows  stakeholders  from  all  sides  to  see  the  number  of 
comments,  where  they  are  in  the  adjudication  process,  and  when  stakeholders  feel 
the  review  is  ready  to  be  closed. 

8 

The  program  had  metrics  set  to  meet  threshold  and  objective  criteria 

9 

When  it  came  to  combining  reviews  or  merging  them,  then  yes.  In  the  past  we  have 
had  several  reviews  skip  the  PDR  and  go  to  CDR  due  to  multiple  reasons.  The  view 
was  that  it  would  save  on  schedule  and  allow  the  team  to  progress  to  giving  the 
requested  solution  to  requirements  vice  systematically  assessing  as  we  progress 
through  the  engineering  activities. 

Question  12:  Did  the  program  have  consistent  leadership  throughout  a  complete 
SETR  cycle? _ 
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Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

55.3% 

26 

No 

44.7% 

21 

answered  question 

47 

skipped  question 

1 

Question  13:  Please  provide  any  additional  feedback  regarding  SETR 

implementations. 

1 

Some  of  the  questions  set  up  as  “yes”  and  “no”  questions  only  and  do  not 
include  other  or  alternative  response.  There  were  a  couple  of  questions  that 

I  was  not  able  to  respond  with  either  a  “yes”  or  “no.”  For  question  11,  there 
are  metrics  used  but  I  was  unclear  on  what  was  meant  by  ROI.  For  question 
12,  it  was  unclear  what  was  meant  by  “consistent  leadership”  so  I  was  not 
able  to  respond.  Sorry. 

2 

Not  sure  if  this  is  applicable  to  what  you  are  doing.  Projects  supporting 
various  customers,  we  follow  the  SIPH  MILCON  process. 

3 

SETR  implementation  is  a  good  check  and  balance  to  see  if  processes  in 
program  is  actually  implemented  as  shown  in  progress. 

4 

SETR  in  static  conditions  is  very  good,  but  is  not  suitable  for  a  situation 
where  things  can  change  too  quickly,  where  laws  and  policies  require  ‘now’ 
(immediate)  changes  to  meet  any  authoritative  mandates  dictated. 

5 

The  program  overestimated  the  value  of  the  COTS  product  and  tried  to 
expedite  the  requirements  phase.  The  program  should  have  maintained  a 
separate  Requirements  Review  vice  merging  it  with  Preliminary  Design 
Review. 

6 

SETR  Events  are  usually  of  value  added,  but  in  an  agile  methodology  / 
framework  environments  such  events  become  challenging  as  the 
atmosphere  changes  dynamically  and  with  an  ad-hoc  nature. 

7 

SETR  events  are  not  consistent  throughout  the  organization.  People 
attending  the  events  have  very  little  value  to  the  process. 

8 

It’s  difficult  for  Tech  Warrant  Holders  to  understand  entrance  &  exit 
criteria  for  Business  System  SETR  event  because  there  isn’t  a  real  tailored 
guideline  or  acquisition  SETR  schedule  for  large  business  system  ACAT 
programs. 

9 

Functional  community  reluctant  to  adopt  modern  technology  even  when  it 
was  placed  at  their  finger  tips. 

10 

In  my  experience,  SETR  events  often  become  a  “check  the  box”  sort  of 
event  if  the  program  is  required  to  complete  every  single  SETR  review,  or 
even  to  address  every  single  “bullet  point”  for  a  given  review,  regardless  of 
how  applicable  it  may  be  to  the  given  program  (if  applicable  at  all).  SETR 
reviews  are  absolutely  necessary,  but  forcing  programs  to  complete  reviews 
that  are  extraneous  given  their  specific  situation  leads  to  increased  costs 
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(especially  on  the  part  of  the  contractor,  especially  on  smaller  programs), 
schedule,  etc. 

11 

The  AAPs  fell  under  P.  Reddy  and  were  done  correctly.  The  ACAT-III  can 
use  some  help,  Most  team  members  do  not  understand  the  intent  or  benefits 
of  these  events. 

12 

Need  to  have  a  stick  to  beat  PMs  over  the  head  with  IRT  SETR.  No  SETR 
should  equal  a  very  bad  FITREP  or  something.  There  just  is  no  teeth  to  5.0 
type  work  and  PMs  can  do  what  they  want. ..and  waste  our  tax  dollars. 

13 

Tailoring  SETR  events  has  pros  and  cons.  SETR  events  typically  take  a 
block  of  preparatory  time  and  resolution  time  to  address  RFAs.  This 
typically  results  in  schedule  delays.  The  benefit  is  that  multiple  SETR 
events  ensure  documentation  is  being  updated  as  the  program  matures  vice 
the  documents  becoming  static  and  outdated. 

14 

Even  though  the  program  was  an  AAP,  the  SETR  process  provided 
disciplined  engineering — that  provided  a  product  delivered  on-time — 
however  internal  Navy  interfaces  and  dependencies  caused  delays  to  the 
overall  program.  Those  programs  were  NOT  using  SETR... 

15 

Tailoring  provides  a  costs  savings  but  the  most  difficult  part  is  convincing  a 
PM  that  is  looking  at  a  list  that  something  is  not  needed  or  can  be  a 
simplified  version. 

16 

While  MCSC  has  a  tech  review  handbook,  different  program  offices  seem 
to  handle  reviews  substantially  differently.  Some  program  offices  do  a  two 
week  read  ahead  +  1  day  review  while  others  do  a  two  week  read  ahead  + 
kick-off  +  review  period  +  comment  adjudication  period  +  capstone.  In  the 
latter  case,  tech  reviews  seem  to  stretch  out  for  many  weeks/months.  We’ve 
captured  that  time  in  our  process  moving  forward,  but  8-12  weeks  seems 
excessive  for  a  tech  review. 

17 

We  have  now  moved  to  different  leadership,  with  an  increased  focus  on  the 
administrative  aspect  of  SETRs;  participants  are  expected  to  review 
applicable  documents  outside  of  the  actual  review  event.  That  does  not 
happen — and  assuming  it  will  is  setting  us  up  for  future  issues. 

18 

I  have  none 

19 

It  is  extremely  difficult  to  determine  what  is  required.  Even  obtaining 
examples  of  good  SETR  documentation  seems  impossible.  Because  of  this, 
it  is  difficult  to  feel  prepared  and  reviews  feel  more  like  “gotcha  events” 
than  the  assistance  they  are  supposed  to  be. 

20 

SETR  Is  a  great  method  to  manage  progress  of  programs 

21 

Program  was  post  Milestone  C,  but  not  yet  fielded  when  I  joined  the  team. 
During  my  tenure,  we  achieved  FOC. 

22 

Managing  expectations  &  frequent  sync  sessions  are  critical  to  success.  Ech 

2  &  Ech  3  have  different  interpretations  of  what  is  expected  regarding  the 
level  of  completeness  of  each  SETR.  Often  Ech  2  is  more  concerned  about 
a  check  in  the  box  than  ensuring  the  engineering  products  are  really  useful. 

23 

I  have  the  brief  for  the  Agile/SETR  process  that  programs  have  had  success 
in  using.  However,  not  all  programs  are  allowed  to  use  that  brief.  That  is 
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due  to  external  organization  and  not  ours. 

24 

I  am  referring  to  CANES  and  the  DOT&E  oversight  of  the  program. 

25 

SETR  process  was  a  good  framework  for  tailored  acquisition,  but  still  new 
to  the  external  stakeholders.  This  created  the  need  to  educate  people  more 
on  the  modification  from  the  norm. 

26 

For  a  smaller  demonstration  type  program,  filling  in  the  gaps  created  by 
tailoring  with  the  routine  program  reviews  has  worked  well.  A  lot  of  the 
same  things  are  covered,  but  in  a  more  flexible  construct. 

27 

There  is  often  a  disconnect  between  the  cost  and  schedule  of  a  program  and 
the  goal  of  the  SETR  which  is  more  focused  on  performance.  Performance 
is  often  the  bill  payer  for  cost  and  schedule. 

28 

I  have  been  a  part  of  many  types  of  reviews  over  the  years.  Depending  on 
the  effort,  the  tailoring  was  handled  differently  each  time  (except  for  the 
PDR,  9/10  times  it  is  merged  into  CDR).  I  treat  that  action  as  Leadership’s 
view  to  handle  SETR  reviews  and  a  vision  that  the  workforce  is  really  just 
hardware  integrators  vice  engineers  (no  real  development  effort,  just 
repackage  existing  components). 

29 

While  SETR  is  a  great  tool  to  assess  where  a  program  is  in  the  time  of  the 
event.  We  are  normally  limited  in  time  and  resources  on  how  deep  to 
conduct  the  review.  We  are  also  limited  to  the  information  that  the 
programs  provide. 

I  had  an  instance  where  the  performers  provided  all  the  necessary 
information,  everything  appeared  on  schedule  for  delivery  and  the  SETR 
went  by  smoothly.  I  later  found  out  that  the  performers  were  1 1  months 
behind  schedule,  but  this  was  not  apparent  with  the  provided 
documentation. 

While  SETR  is  a  good  tool,  it  is  often  cumbersome  to  the  programs  to 
support  this  effort  while  still  performing  their  daily  tasks. 

30 

SETR  should  be  a  means  to  implement  engage  users  and  customers  and  vet 
risks  before  they  become  issues. 
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